- Webcast - Oracle Document Cloud Service (DOCS): Technical Overview (Oracle Partner Hub: ISV Migration Center Team)
via Oracle Partner Hub: ISV Migration Center Team https://blogs.oracle.com/imc/
Alfresco support, by default, the preview in your browser of some video formats like mp4 but doesn't support some others like mkv, avi or wmv. Even if Google Chrome can read mkv files for example, if you try to use the action "View in Browser" (this is not the same thing as the preview), Chrome will not try to play the mkv file but will download it instead. That's why if you upload an mp4 video in Alfresco, you will certainly be able to watch this video directly in your browser. In this blog, I will try to explain how to configure an external tool to take care of the thumbnails creation and transformations for your videos. To achieve that, we will install and configure ffmpeg. In addition to that, you can also very easily configure Alfresco to embed an external video player like "FlowPlayer" that would take care of playing all video formats directly "streaming" from Alfresco, not using your browser. Basically, this is done by replacing the preview page for some Mime types but I will not describe it in details here.
I. Installation of ffmpeg
For this blog post, I will use /opt/alfresco-4.2.c as the $ALF_HOME environment variable: the folder where Alfresco has been installed. The current release of ffmpeg is version 2.6.3. This is at least working for Alfresco v4.2.x and v5.x.
The ffmpeg binaries/executables are available in the ffmpeg website. From this website, download the "Static" build for Windows, Linux or Mac depending on the Operating System on which Alfresco has been installed. For this example, I will use a linux host (RedHat 6.x, 64 bits) but I can assure you that it's working properly on Windows too. I already installed ffmpeg with several versions of Alfresco from v4.2.x to 5.x.
[alfresco ~]$ cd /opt
[alfresco /opt]$ wget http://johnvansick...elease-64bit-static.tar.xz
[alfresco /opt]$ tar -xJf ffmpeg-release-64bit-static.tar.xz
[alfresco /opt]$ chmod -R 755 ffmpeg-2.6.3-64bit-static
[alfresco /opt]$ chown -R alfresco:alfresco ffmpeg-2.6.3-64bit-static
[alfresco /opt]$ mv ffmpeg-2.6.3-64bit-static /opt/alfresco-4.2.c/ffmpeg
II. Configuration of Alfresco
These commands will download, extract, change the permissions/ownership and rename the folder to something more friendly (at a more proper location too).
Once ffmpeg has been installed, the next step is to configure Alfresco to use it. The first thing to do is to add some parameters in the well-known alfresco-global.properties file. Don't hesitate to customize these paremeters, remove the lines that aren't needed, aso...
[alfresco /opt]$ cd /opt/alfresco-4.2.c/tomcat/shared/classes
[alfresco /classes]$ cat alfresco-global.properties
### Begin of the file with your custom parameters ###
### E.g.: dir.root, db.driver, db.username, aso... ###
### FFMPEG executable location ###
### Video Thumbnails parameters ###
### Video Transformations parameters ###
In the above configuration, the parameter "ffmpeg.exe=/opt/alfresco-4.2.c/ffmpeg/ffmpeg" is the location of the binary file named ffmpeg. If you install Alfresco and ffmpeg on a Windows host, then this parameter looks like: "ffmpeg.exe=C:/Alfresco-4.2.c/ffmpeg/bin/ffmpeg.exe"
Once this is done, you need to enable the thumbnail & transformation extensions in Alfresco and this is done using the following steps:
[alfresco /classes]$ cd /opt/alfresco-4.2.c/tomcat/shared/classes/alfresco/extension
[alfresco /extension]$ mv video-thumbnail-context.xml.sample video-thumb-context.xml
[alfresco /extension]$ mv video-transformation-context.xml.sample video-transf-context.xml
If you want, you can parse these two files to understand what Alfresco will do with ffmpeg but basically these files define the commands and options that will be used by Alfresco.
And that's it, Alfresco has been configured to use ffmpeg for thumbnails creation and transformations of your video. For Alfresco to take these changes into account, simply restart your Application Server using an Alfresco service:
[alfresco ~]$ service alfresco restart
Or using Alfresco default scripts:
[alfresco ~]$ /opt/alfresco-4.2.c/alfresco.sh stop
[alfresco ~]$ /opt/alfresco-4.2.c/alfresco.sh start
To check if ffmpeg is working properly, simply open Alfresco Share, browse your repository to find a video with an avi or wmv format and a thumbnail should now be present (if not, just refresh the page...). You can also try to upload a video and if a thumbnail is created for this video, then ffmpeg is working!
An Oracle employee sent me an email today, inquiring about the Open Mic Night. This employee, who is a user of Oracle Application Express at Oracle, will be attending the Kscope conference for the very first time. As I replied to him in email:
Open Mic night will be on Monday evening, from 8:00P - 10:00P. You would think that most people would call it a day (after a long day), but it's usually a packed room.
Open Mic night is the attendee's night to shine in front of their fellow attendees. People are given roughly 5 - 10 minutes to show off what they've done with APEX - it's timed. No PPT. If you show a PowerPoint, you will be booed. You're on stage, you plugin your laptop to a projector, and you present on a big screen. It's just a great way for people in the #orclapex community to proudly show what they've accomplished. I've seen some extraordinarily creative and professional solutions from our customers.
The time goes by fast, so you have to come prepared. And the Oracle APEX team usually sponsors the beer for this event, so it can get a bit rowdy. ;)
If you're at Kscope15 for the APEX track, or even half-curious about APEX, it's a "must attend" event.
Here's a shot from last year's Open Mic Night:
Installation was not bad.
Evodesk is the 2nd company I am aware of for using recesss nuts for desktop installation.
This makes much more sense than using wood screws.
However, pondering if Evodeskk will be the first company to used recess nuts for all components and get rid of wood screws.
Don’t you think the legs make great towel rack?
This Log Buffer edition covers Oracle, SQL Server and MySQL blog posts from all over the blogosphere!
Hey DBAs: You know you can install and run Oracle Database 12c on different platforms, but if you install it on an Oracle Solaris 11 zone, you can take additional advantages.
Here is a video with Oracle VP of Global Hardware Systems Harish Venkat talking with Aaron De Los Reyes, Deputy Director at Cognizant about his company’s explosive growth & how they managed business functions, applications, and supporting infrastructure for success.
Oracle Unified Directory is an all-in-one directory solution with storage, proxy, synchronization and virtualization capabilities. While unifying the approach, it provides all the services required for high-performance enterprise and carrier-grade environments. Oracle Unified Directory ensures scalability to billions of entries. It is designed for ease of installation, elastic deployments, enterprise manageability, and effective monitoring.
Understanding Flash: Summary – NAND Flash Is A Royal Pain In The …
Extracting Oracle data & Generating JSON data file using ROracle.
It is no good doing some or most of the aspects of SQL Server security right. You have to get them all right, because any effective penetration of your security is likely to spell disaster. If you fail in any of the ways that Robert Sheldon lists and describes, then you can’t assume that your data is secure, and things are likely to go horribly wrong.
How does a column store index compare to a (traditional )row store index with regards to performance?
Learn how to use the TOP clause in conjunction with the UPDATE, INSERT and DELETE statements.
Did you know that scalar-valued, user-defined functions can be used in DEFAULT/CHECK CONSTRAINTs and computed columns?
Tim Smith blogs as how to measure a behavioral streak with SQL Server, an important skill for determining ROI and extrapolating trends.
Pilip Horan lets us know as How to run SSIS Project as a SQL Job.
Encryption is important component of secure environments. While being intangible, property security doesn’t get enough attention when it comes to describing various systems. “Encryption support” is often the most of details what you can get asking how secure the system is. Other important details are often omitted, but the devil in details as we know. In this post I will describe how we secure backup copies in TwinDB.
The fsfreeze command, is used to suspend and resume access to a file system. This allows consistent snapshots to be taken of the filesystem. fsfreeze supports Ext3/4, ReiserFS, JFS and XFS.
Shinguz: Controlling worldwide manufacturing plants with MySQL.
MySQL 5.7.7 was recently released (it is the latest MySQL 5.7, and is the first “RC” or “Release Candidate” release of 5.7), and is available for download
Upgrading Directly From MySQL 5.0 to 5.6 With mysqldump.
One of the cool new features in 5.7 Release Candidate is Multi Source Replication.
By Phil HillMore Posts (318)
I answered another question by saying that the LMS, with multiple billions invested over 17+ years, has not “moved the needle” on improving educational results. I see the value in providing a necessary academic infrastructure that can enable real gains in select programs or with new tools (e.g. adaptive software for remedial math, competency-based education for working adults), but the best the LMS itself can do is get out of the way – do its job quietly, freeing up faculty time, giving students anytime access to course materials and feedback. In aggregate, I have not seen real academic improvements directly tied to the LMS.
In response, Mark gives “a personal view of my own journey towards LMS nihilism” in a post titled “How I lost my faith in the LMS” that has some excellent points (first go read his whole post, I’ll wait).
Mark describes how the LMS market in Australia changed dramatically – mostly towards Moodle – due to Bb / D2L lawsuit, end-of-life for WebCT, and Moodle release of 1.9, noting:
There were, I believe, a variety of reasons that Moodle was so successful during this time, but one of the most common things that I would hear during this period was that, compared to incumbent LMS, Moodle simply ‘got out of the way’ and let academic staff do their thing. It helped the LMS stop being a barrier, and moved it closer to being an enabler, which is exactly what it should have been.
During this time Moodle was booming in popularity, and the transitions I was involved in by and large went as well as any other campus-wide technology platform change can, but one big question (and I must send out a thank you my friend and sounding board James Hamilton for planting this seed) was lurking in the background – how do we measure the success of the implementation? How do we know that the LMS in and of itself is making any difference whatsoever?
The answer he comes to is that no, the LMS in and of itself does not change outcomes and that:
The specific LMS that was in use paled into insignificance next to the innovation, dedication and craftiness of the person using it.
Here Mark makes one of the best points I’ve seen in the LMS discussions of late [emphasis added].
In a commodity market, the argument often turns to cost. In the case of the LMS, like any piece of campus-wide technology, the cost of the service in technology terms often pales into insignificance when compared with the cost in terms of time spent (wasted?) by academic and administrative staff being forced to use a system designed to try and satisfy a large set of complex requirements. Perhaps this was one of the most compelling things about Moodle back in its heyday – the perception that it simply ‘got out of the way’ of teachers wanting to do their job – and the significant ‘switching cost’ in terms of managing a large-scale change program that is needed to swap out an LMS was deemed worth it in terms of the longer term reduction of burden on users.
That was then, however, not now.
Where we slightly differ is in the conclusions. The lack of evidence of LMS usage directly impacting academic results does not make the LMS a commodity and is no reason to go nihilist.
Two measures of value in a traditional LMS can be thought of as how well it ‘gets out of the way’ and how well it enables apps that can directly affect student learning. From my experience, the various LMS options differ greatly in these two attributes. I have seen examples at campuses where an LMS adoption led to one that was much more intuitive, reliable, and easy to adopt to the point that training resources were diverted away from ‘here’s how to migrate a course and which button to push’ to ‘here are some pedagogical improvements to consider using online tools’. I have seen schools benefit simply from having reliable systems that don’t go down during exams. In other words, and LMS solution can significantly reduce the “cost in terms of time spent by academic and administrative staff”. And by the way, that choice might not always be the same LMS – it depends greatly on course design and pedagogical models.
While I detest most RFP processes, there are examples (typically involving creative compliance with purchasing rules or active support from an enlightened purchasing guru) where the planning process itself leads to increased collaboration among academic and administrative staff. If done well, a vendor selection process can enable greater focus on teaching and learning effectiveness and cross-pollination of ideas.
Update (hit publish too soon): I have also seen situations where an LMS is so painful to use that faculty don’t take advantage of tools that are appropriate or useful. While there is risk in broadly looking at depth of LMS adoption as a net positive, the wrong LMS choice or implementation can prevent faculty or instructional designers from doing what they’d like.
While I might be misreading Mark’s nihilism reference, he makes some great observations based on his personal journey. In the end, however, I do not see the LMS as a commodity.
- Mark’s conclusion is “So then, in my mind, while the LMS may not quite yet be considered a commodity in terms of features and functions, it might as well be a commodity in terms of the overall impact it has on student learning outcomes.
The post LMS Observations: You had me until you went nihilist appeared first on e-Literate.
We share our skills to maximize your revenue!
I have just spent an hour or so working on getting a sample scheduleReport web service working with parameter values. There are a lot of examples out there but none I have found have the parameters being set. Our doc is a little light on details on how to set them up :) In lieu of that, here's this!
// Set the parameter values for the report. In this example we have // 'dept' and 'emp' parameters. We could easily query the params dynamically //Handle 'dept' parameter ParamNameValue deptParamNameVal = new ParamNameValue(); deptParamNameVal= new ParamNameValue(); deptParamNameVal.setName("dept"); // Create the string array to hold the parameter value(s) ArrayOfXsdString deptVal = new ArrayOfXsdString(); // For individual values or multiples, add values to the // string array e.g. 10,20,30 deptVal.getItem().add("10"); deptVal.getItem().add("20"); deptVal.getItem().add("30"); // Asterisk used for a null value ie 'All' //deptVal.getItem().add("*"); // add the array to the parameter object deptParamNameVal.setValues(deptVal); //Handle 'emp' parameter ParamNameValue empParamNameVal = new ParamNameValue(); empParamNameVal= new ParamNameValue(); empParamNameVal.setName("emp"); ArrayOfXsdString empVal = new ArrayOfXsdString(); // For individual values or multiples, add values to the string array // empVal.getItem().add("Jennifer Whalen"); // empVal.getItem().add("Michael Hartstein"); // Asterisk used for a null value ie 'All' empVal.getItem().add("*"); empParamNameVal.setValues(empVal); // add parameter values to parameter array ArrayOfParamNameValue paramArr = new ArrayOfParamNameValue(); paramArr.getItem().add(deptParamNameVal); paramArr.getItem().add(empParamNameVal); //Now add array to values obj ParamNameValues pVals = new ParamNameValues(); pVals.setListOfParamNameValues(paramArr);
The pVals object can then be added to the report request object.
This is very cool:
There are a bunch of nice computer science classes online from MIT: Free online MIT computer science classes
Here is an introductory computer science class: Intro to computer science
Here is a graded version of the same class on edX: Graded version of MIT intro CS class starting June 10th.
edX does not have as many computer science classes but edX may motivate students because edX classes include grades and certificates for those who pass.
I use computer science every day in my database work but I have not taken a formal class since 1989.
I have been on a computer science kick ever since watching The Imitation Game. I downloaded Turing’s 1936 paper after watching the movie. I got about halfway through it before giving up. It was dense! Maybe will take another stab at it some day. But, the MIT classes are neat because they are the way computer science is now taught, and hopefully they are easier to understand than Turing’s paper.
By Phil HillMore Posts (318)
In the discussions at Google+ based on last week’s post about the Miami University of Ohio disability discrimination lawsuit, George Station made two important points that deserve more visibility.
It’s been a-coming for several years now. Cal State has some pretty strong rules in place for compliance with ADA and state-level disability laws. Still, [Universal Design for Learning] UDL is a little-known acronym on any campus you care to visit, and staff support is probably one person in an office, except for Miami of Ohio as of this week, I guess…
Add the recent edX settlement with the US Department of Justice, and the whole direction of edtech changes…
Put another way, it should come as no surprise that the US Department of Justice is ramping up its enforcement of disability discrimination regulations in the education world. Captioning service provider CaptionSync has an excellent summary of the field, written before the DOJ intervention at Miami.
Accessibility laws applicable to higher education have been in place in the United States for decades, but many schools are still not fully compliant with the laws. Part of the lag in compliance can be attributed to lenient enforcement in the early years of these laws; the Rehabilitation Act was enacted in 1973 and the Americans with Disabilities Act was enacted in 1990, but initially there were very few government investigations or enforcement actions. Over time both government agencies (such as the Office for Civil Rights) and advocacy groups (such as the National Federation for the Blind and the National Association for the Deaf) have increasingly been making efforts to enforce the provisions of these laws. Recent civil suits filed by the National Association for the Deaf (NAD) and other advocacy organizations against both Harvard and MIT suggest that now is a good time to take a hard look at your accessibility compliance efforts if you work with video in a college or university setting.
The Department of Justice (DOJ) sent a letter to all college and university presidents on the topic of accessibility for emerging technologies in 2010; it contained a useful summary of various accessibility regulations and how they apply to the education community.
In February, the National Association of the Deaf (NAD) filed suit against Harvard and MIT based on their MOOCs using edX. It is worth noting that the lawsuit is against the schools, not the MOOC provider. In the announcement:
Many videos simply aren’t captioned at all. For example, a Harvard program on the 50th anniversary of Brown v. Board of Education, a 2013 Harvard Q&A with Bill Gates and a 2013 MIT discussion with MIT professor Noam Chomsky about the leaks attributable to Chelsea (formerly Bradley) Manning all lack closed captions.
“Worse still,” said attorney Timothy Fox, “a sampling of the videos available illustrates the problem with inaccurate captioning, making them confusing and sometimes completely unintelligible.”
The issue is not that there is no capability for captioning, but that those producing the content (Harvard and MIT) do not provide captions or do some with many errors. Subsequently, the DOJ and edX settled out of court based on the following:
5. Following the compliance review, the United States determined that www.edx.org and the Platform were not fully accessible to some individuals with disabilities in violation of Title III of the ADA.
6. EdX disputes the findings set forth above and denies that www.edx.org, its mobile applications, and the Platform are covered by or are in violation of Title III of the ADA.
In the settlement, both parties go out of their way to clarify that edX is a software provider and that the schools are content providers. The DOJ settlement calls on edX within 18 months to conform with the Web Content Accessibility Guidelines (“WCAG”) 2.0 AA, published by the Web Accessibility Initiative of the World Wide Web Consortium (“W3C”). More importantly, however, the agreement stipulates that edX within 90 days provide guidance to content providers (schools).
27. Develop a guide for Content Providers entitled Accessibility Best Practices Guidance for Content Providers (“Accessibility Best Practices Guidance”) and distribute a copy to each Content Provider with instructions for redistribution among individuals involved in producing Course Content. The Accessibility Best Practices Guidance shall describe steps and resources on how Course Content may be made to conform with WCAG 2.0 AA for Participants with disabilities using the CMS and inform Content Providers that the following resources may assist them in producing accessible Course Content: UAAG 1.0, ATAG 2.0, WAI-ARIA, WCAG2ICT, EPUB3, DAISY, and MathML.
The DOJ insists not only that software include capabilities for accommodation of students with disabilities but also that schools actually include the content and related metadata that is required for compliance. It is no longer enough for schools to buy software that is “ADA compliant”. Faculty or instructional designers need to include captions, alt-texts and alternate pathways for students to have equal access.
The origin of the Miami U lawsuit and the DOJ intervention is based on blind students, but the issues are the same. Repeatedly the DOJ referred to edtech “as implemented by Miami University”. As noted by reader Brian Richwine, the original lawsuit does reference Sakai (the LMS at the time of lawsuit), but the focus is still on how the content was provided.
Going back to the CaptionSync blog post:
Some schools have pointed out that in the summer of 2015 the DOJ is expected to release new guidance on how accessibility for websites is to be handled and they are awaiting that guidance before they step up their accessibility efforts.
The big lesson is that higher education institutions themselves had better get ready to understand their role as content providers that must conform to disability standards. Just letting individual faculty members figure out what to do is a recipe for future lawsuits. Faculty need support, guidance and (gasp) appropriate oversight to get this right.
Beyond the regulations and frameworks listed in the DOJ documents, schools should also increase their understanding and use of the UDL framework and guidelines that George referenced in his comments.
The goal of education in the 21st century is not simply the mastery of content knowledge or use of new technologies. It is the mastery of the learning process. Education should help turn novice learners into expert learners—individuals who want to learn, who know how to learn strategically, and who, in their own highly individual and flexible ways, are well prepared for a lifetime of learning. Universal Design for Learning (UDL) helps educators meet this goal by providing a framework for understanding how to create curricula that meets the needs of all learners from the start.
The UDL Guidelines, an articulation of the UDL framework, can assist anyone who plans lessons/units of study or develops curricula (goals, methods, materials, and assessments) to reduce barriers, as well as optimize levels of challenge and support, to meet the needs of all learners from the start. They can also help educators identify the barriers found in existing curricula.
- Insert joke here about G+ and its hundreds of active users.
The post Miami, Harvard and MIT: Disability discrimination lawsuits focused on schools as content providers appeared first on e-Literate.
First, I've pushed the limits of my iPad's usability. I really have...for a couple of years. But I just don't find myself using it for any kind of serious work. My MacBook Air has really taken over for both premise-based and cloud-based work. Even with a keyboard of the tablet, I find myself much more productive on the Air.
Second, I'm finally upgrading my iPhone 5 next month. The most likely upgrade candidate is the iPhone 6 Plus. Yes, the Samsung Droid offerings look really great - even better in many respects. But I'm invested in the Apple platform and there's not enough extra benefit in other platforms to justify a transition in my mind. So it's likely the 6 Plus...with a very large screen. And I see the 6 Plus taking over the few functions I now perform with my iPad. So the iPad would become just another weight to lug through the many airports I visit every year. I'm actually looking forward to cutting back on the amount of tech gear carried when I eliminate the iPad.
In all fairness, I think I would have come to this conclusion with any tablet. I still consider the iPad to be the best of the bunch. I have just come to the conclusion that the tablet format just does not provide enough value for me...at least, not right now.
What's to become of the iPad? It may become a third screen in my home office. Or it may become a media viewer on a planned treadmill. I mean, it's nice and all...but, at least for me, I've discovered the iPad concept to be much more of a nice to have than a got to have. Cool but not necessary.
What about you? Different view or experience? Sound off in the comments.
Is your organization using an Oracle product to help with a sustainability initiative while reducing costs? Saving energy? Saving gas? Saving paper?
For example, you may use Oracle’s Agile Product Lifecycle Management to design more eco-friendly products, Oracle Cloud Solutions to help drive down power consumption, Oracle Transportation Management to reduce fleet emissions, Oracle Exadata Database Machine to decrease power and cooling needs while increasing database performance, Oracle Environmental Accounting and Reporting to measure environmental impacts, or one of many other Oracle products.
Your organization may be eligible for the 2015 Oracle Excellence Award: Sustainability Innovation.
Submit a nomination form located here by Friday June 19 if your company is using any Oracle product to take an environmental lead as well as to reduce costs and improve business efficiencies using green business practices. These awards will be presented during Oracle OpenWorld 2015 (October 25-29) in San Francisco.
About the Award
- Winners will be selected from the customer and/or partner nominations. Either a customer, their partner, or Oracle representative can submit the nomination form on behalf of the customer.
- Winners will be selected based on the extent of the environmental impact they have had as well as the business efficiencies they have achieved through their combined use of Oracle products.
- Your company uses at least one component of Oracle products, whether it's the Oracle database, business applications, Fusion Middleware, or Oracle Sun servers/storage.
- This solution should be in production or in active development.
- Nomination deadline: Friday June 19, 2015.
Benefits to Award Winners
- Award presented to winners during Oracle OpenWorld by Jeff Henley, Oracle Executive Vice Chairman of the Board
- 2015 Oracle Sustainability Innovation Award logo for inclusion on your own website and/or other marketing materials
- Possible placement in Oracle Profit Magazine and/or Oracle Magazine
See last year's winners here.
Questions? Send an email to: firstname.lastname@example.orgFollow Oracle’s Sustainability Solutions on Twitter, LinkedIn, YouTube, and the Sustainability Matters blog.
Part III – There’s Something (Profitable) About Your Privacy
Why do so many companies hanker after apps? Smartphone apps, tablet apps, iOS apps, Android apps, app-this, app-that….
Leave aside for a moment the techno-pubescent excitement that accompanies the launch of every new technology (if you are not old enough to remember words like “client-server”, then “soa” will surely sound familiar enough). Every Marketing 101 course drills into its students that acquiring a new customer is way costlier than retain an existing. Loyal customers (leaving aside the pejorative connotation the word “loyal” carries, implying that customers who shop elsewhere for a better deal are of dubious moral character) are what you should aspire to – that keep buying from you for a longer period of time – and which allows you to refocus your marketing and advertising dollars towards the acquisition of newer customers, faster. If you spend less on unnecessary discounts and expensive retention schemes then margins from existing customers are automatically higher.
Customers can stay loyal if you can build a bond of affinity with them. You should aspire to be more like the local kirana owner (only infinitely richer), who in a perfect world knew everything about you – your likes, dislikes, which festivals you celebrated, and therefore which sweets you would buy, when your relatives came over to stay and what their likes were, what exotic food items you wanted, and so on. And who knew your name. Hence the marketer’s love for loyalty programs, no matter that customer loyalty is notoriously difficult to guarantee.
In the world of online retailing (actually, it applies just as well to any kind of retailing), how do you get to acquire a deep level of intimacy with your customer? Smartphone apps provide this degree of intimacy that desktop / laptop browsers cannot. This is by simple virtue of the fact that the smartphone travels with the user, the user is constantly logged on to the app, and the app knows where you go and where you are. So no wonder that in December 2011, Amazon offered a “brazen” deal to its customers in brick-and-mortar stores to do an “in-store” price-check of items using the Amazon Price Check app, and if the same product was available on Amazon, get it at a discount off the store’s price. Though termed “not a very good deal”, it nonetheless angered the Retail Industry Leaders Association, and elsewhere was described as “Evil But It's the Future”. The combination of availability – the app was installed on the smartphone that was with the user – and the integrated capabilities in the device – a camera that fed into a barcode scanner app –made this possible. The appeal of apps is undeniable.
The magical answer is – “app”. Your best-thing-since-sliced-bread app is installed on the customer’s smartphone (or tablet or phablet), is always running (even when it is not supposed to be running), knows everyone in your contacts (from your proctologist to the illegal cricket bookie), can hear what you speak (even your TV can do this now), knows where you are, who you call, what text messages you send and receive, knows what other apps you have installed on your smartphone (presumably so it can see how potentially disloyal you could be), which Wi-Fi networks you connect to, access what photos and videos you have taken (naughty!) and so on and so forth. All this the better to hear you with, the better to see you with, and ultimately the better to eat you (your wallet) with – with due apologies to Little Red Riding Hood. You may want to take a closer look at the permissions your favorite app wants when you install it – like Amazon India, eBay, Flipkart, Freecharge, HomeShiop18, Jabong, MakeMyTrip, Myntra, SnapDeal. Great minds do seem to think alike, don’t they?
[Technical aside: I covered the red herrings thrown in favour of apps in the first part, but here is some more… You can store more data, more effectively, and process that data better using an app than you can with a plain browser-based approach. True. But not quite. The ever-evolving world of HTML5 (the standard that underpins how information is structured and presented on the web) has progressed to make both these points moot – with offline storage and local SQL database support. Yes, there are arguments to be made about handling large amounts of data offline with browser-based mechanisms, but these are for the most part edge-cases. To be fair, there are some high-profile cases of companies switching to native apps after experimenting with HTML5-based apps (hybrid apps that wrapped a browser-based UI with a native shell), like LinkedIn and Facebook. The appeal of apps therefore is undeniable. But, as I argued earlier, the appeal of apps does not negate the utility of browser-based interfaces.]
What is all this useful for? Your app now knows that you Ram, Shyam, and Laxman in your contacts have birthdays coming up, and it can suggest an appropriate gift for them. Convenient, isn’t it? While driving to work, you can simply tell your app – speak out the commands – to search for the latest perfume that was launched last week and to have it gift wrapped and delivered to your wife. The app already has your credit card details, and it knows your address. Your app knows that you are going on a vacation next week (because it can access your calendar, your SMS-es, and perhaps even your email) to Sikkim; it helpfully suggests a wonderful travel book and some warm clothing that you may need. The imagined benefits are immense.
But, there is a distinctly dark side to apps – as it relates to privacy – that should be a bigger reason of concern for customers and smartphone users alike. Three sets of examples should suffice.
You get a flyer from your favourite brick-and-mortar store, letting you know that you can buy those items that your pregnant daughter will need in the coming weeks. You head over to the store, furious – because your daughter is most certainly not pregnant. Later you find out that she is, and that the store hadn’t made a mistake. It turns out the truth is a little more subtler than that, and a little more sedate than what tabloid-ish coverage - with headlines like “How Companies Learn Your Secrets” - made it out to be (the original presentation made at the PAW Conference is also available online).
There are enough real dangers in this world without making it easier to use technology to make it even more unsafe. Considering how unsafe air travel can be for women and even girls, one has to question the wisdom of making it even more so. If this does not creep you out, then perhaps the Tinder app – which uses your location and “displays a pile of snapshots of potential dates in a user’s immediate area”, to as close as within 100 feet - may give you pause for thought.
Do apps need all the permissions they ask for? No. But, … no! Would they work if they didn’t have all those permissions? 99% of the time, yes – they would work without a problem. For example, an app would need to access your camera if you wanted to scan a barcode to look up a product. The app would need access to your microphone if you wanted to speak out your query rather than type it in the app. What if you don’t particularly care about pointing your camera at the back of books to scan their barcodes, or speaking like Captain Kirk into your phone? Sorry, you are out of luck. You cannot selectively choose to not grant to certain privileges to an app – at least on a device running the Android mobile operating system. In other words, it is a take-it-or-leave-it world, where the app developer is in control. Not you. And wanting to know your location? Even if you are a dating app, it’s still creepy.
But surely app makers will ask you before slurping your very personal, very private information to its servers in the cloud? Yes, of course – you believe that to be true, especially if you are still in kindergarten.
A few weeks before its IPO, JustDial’s app was removed from the Google Play Store. It was alleged that the updated version of the JustDial app had “started retrieving and storing the user’s entire phone book, without a warning or disclaimer. ,” Thereafter, JustDial’s mobile “Terms and Conditions” were updated to include the following line: “You hereby give your express consent to Justdial to access your contact list and/or address book for mobile phone numbers in order to provide and use the Service.”
In 2013, US-based social networking app Path was caught as it “secretly copied all its users’ iPhone address books to its private servers.” Action was swift. The FTC investigated and reached a settlement with Path, which required “Path, Inc. to establish a comprehensive privacy program and to obtain independent privacy assessments every other year for the next 20 years. The company also will pay $800,000 to settle charges that it illegally collected personal information from children without their parents’ consent.” In the US, a person’s address book “is protected under the First Amendment.” When the controversy erupted, it was also reported that “A person’s contacts are so sensitive that Alec Ross, a senior adviser on innovation to Secretary of State Hillary Rodham Clinton, said the State Department was supporting the development of an application that would act as a “panic button” on a smartphone, enabling people to erase all contacts with one click if they are arrested during a protest.” Of course, politics is not without its dose of de-rigueur dose of irony. That dose was delivered in 2015 when it emerged that Hillary Clinton had maintained a private email account even as she was Secretary of State in the Barack Obama presidency and refused to turn over those emails.
So what happened to Just Dial for allegedly breaching its users’ privacy? Nothing. No investigation. No fine. No settlement. No admission. No mea-culpa. In short, nothing. It was business as usual.
Apps can be incredibly liberating in eliminating friction in the buying process. But hitching your strategy to an app-only world is needless. It is an expensive choice – from many, many perspectives, and not just monetary. The biggest costs are of making you look immature should you have to reverse direction. As a case-in-point, one can point to the entirely avoidable brouhaha over Flipkart, Airtel, and Net Neutrality. In this battle, no one came smelling like roses, least of all Flipkart, which attracted mostly negative attention from the ill-advised step, notwithstanding post-fact attempts to bolt the stable door.
Let me end with an analogy. The trackpad on your laptop is very, very useful. Do you then disable the use of an externally connected mouse?
Disclaimer: views expressed are personal.
 "Computerworld - Google Books", https://books.google.co.in/books?id=c2t_-WWE1VAC&pg=PA109&lpg=PA109&dq=client-server+hype&source=bl&ots=SJGHWFM-M5&sig=g6sagoJV_xVSvp22-rgOonfLpNY&hl=en&sa=X&ei=x1ExVb3NKYeumAX2_IGICQ&ved=0CDYQ6AEwBA#v=onepage&q=client-server%20hype&f=false
 "SOA: Hype vs. Reality - Datamation", http://www.datamation.com/entdev/article.php/3671061/SOA-Hype-vs-Reality.htm
 "How Valuable Are Your Customers? - HBR", https://hbr.org/2014/07/how-valuable-are-your-customers/
 "Loyalty programmes: Are points that consumers stockpile juicy enough to keep them coming back? - timesofindia-economictimes", http://articles.economictimes.indiatimes.com/2013-06-30/news/40272286_1_loyalty-programmes-loyalty-card-loyalty-management
 "What Loyalty? High-End Customers are First to Flee — HBS Working Knowledge", http://hbswk.hbs.edu/item/6679.html
 "Amazon's Price Check App Undercuts Brick-and-Mortar Stores Prices | TIME.com", http://business.time.com/2011/12/08/use-amazons-price-check-app-and-save-15-this-saturday/
 "Amazon.com Help: About the Amazon Price Check App", http://www.amazon.com/gp/help/customer/display.html?nodeId=200777320
 "Amazon pushing Price Check app with controversial online discounts | The Verge", http://www.theverge.com/2011/12/10/2626703/amazon-price-check-app-competition-discount
 "Retail association pissed about Amazon.com's Price Check app - GeekWire", http://www.geekwire.com/2011/retail-association-pissed-amazoncoms-price-check-app/
 "Amazon Price Check May Be Evil But It's the Future - Forbes", http://www.forbes.com/sites/erikkain/2011/12/14/amazon-price-check-may-be-evil-but-its-the-future/
 "Samsung smart TV issues personal privacy warning - BBC News", http://www.bbc.com/news/technology-31324892
 "Little Red Riding Hood - Wikipedia, the free encyclopedia", http://en.wikipedia.org/wiki/Little_Red_Riding_Hood
 "Web Storage", http://dev.w3.org/html5/webstorage/
 "Offline Web Applications", http://www.w3.org/TR/offline-webapps/#sql
 "Why LinkedIn dumped HTML5 & went native for its mobile apps | VentureBeat | Dev | by J. O'Dell", http://venturebeat.com/2013/04/17/linkedin-mobile-web-breakup/
 "Mark Zuckerberg: Our Biggest Mistake Was Betting Too Much On HTML5 | TechCrunch", http://techcrunch.com/2012/09/11/mark-zuckerberg-our-biggest-mistake-with-mobile-was-betting-too-much-on-html5/
 "Did Target Really Predict a Teen’s Pregnancy? The Inside Story", http://www.kdnuggets.com/2014/05/target-predict-teen-pregnancy-inside-story.html
 "How Companies Learn Your Secrets - NYTimes.com", http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html?_r=0
 "Predictive Analytics World Conference: Agenda - October, 2010", http://www.predictiveanalyticsworld.com/dc/2010/agenda.php#day1-8a
 "Federal judge upholds verdict that North Bergen man molested woman on flight ‹ Cliffview Pilot", http://cliffviewpilot.com/federal-judge-upholds-verdict-that-north-bergen-man-molested-woman-on-flight/
 "Man accused of groping woman on flight to Newark - NY Daily News", http://www.nydailynews.com/new-york/man-accused-groping-woman-flight-newark-article-1.1709952
 "Man jailed for molesting girl, 12, on flight to Dubai | The National", http://www.thenational.ae/uae/courts/man-jailed-for-molesting-girl-12-on-flight-to-dubai
 "Virgin is Going to Turn Your Flight Into a Creepy Bar You Can't Leave", http://mic.com/articles/37807/virgin-is-going-to-turn-your-flight-into-a-creepy-bar-you-can-t-leave
 "KLM Introduces A New Way To Be Creepy On An Airplane - Business Insider", http://www.businessinsider.com/klm-introduces-a-new-way-to-be-creepy-on-an-airplane-2012-2?IR=T
 "Tinder Dating App Users Are Playing With Privacy Fire - Forbes", http://www.forbes.com/sites/anthonykosner/2014/02/18/tinder-dating-app-users-are-playing-with-privacy-fire/
 "Include Security Blog | As the ROT13 turns….: How I was able to track the location of any Tinder user.", http://blog.includesecurity.com/2014/02/how-i-was-able-to-track-location-of-any.html
 http://en.wikipedia.org/wiki/Justdial, accessed April 11, 2015
 "Updated: JustDial App Pulled From Google Play Store; Privacy Concerns? - MediaNama", http://www.medianama.com/2012/09/223-justdial-app-pulled-from-google-play-store-privacy-concerns/
 "Updated: JustDial App Pulled From Google Play Store; Privacy Concerns? - MediaNama", http://www.medianama.com/2012/09/223-justdial-app-pulled-from-google-play-store-privacy-concerns/
 "Bad App Reviews for Justdial JD", http://www.badappreviews.com/apps/147872/justdial-jd-search-anything, accessed April 09, 2015
 "The Path Fiasco Wasn't A Privacy Breach, It Was A Data Ownership Breach - The Cloud to Cloud Backup Blog", http://blog.backupify.com/2012/02/09/the-path-fiasco-wasnt-a-privacy-breach-it-was-a-data-ownership-breach/
 "Path Social Networking App Settles FTC Charges it Deceived Consumers and Improperly Collected Personal Information from Users' Mobile Address Books | Federal Trade Commission", https://www.ftc.gov/news-events/press-releases/2013/02/path-social-networking-app-settles-ftc-charges-it-deceived
 "Anger for Path Social Network After Privacy Breach - NYTimes.com", http://bits.blogs.nytimes.com/2012/02/12/disruptions-so-many-apologies-so-much-data-mining/?_r=0
 "Hillary Clinton deleted 32,000 'private' emails, refuses to turn over server - Washington Times", http://www.washingtontimes.com/news/2015/mar/10/hillary-clinton-deleted-32000-private-emails-refus/
 "Flipkart Pulls Out of Airtel Deal Amid Backlash Over Net Neutrality", http://www.ndtv.com/india-news/flipkart-pulls-out-of-airtel-deal-amid-backlash-over-net-neutrality-754829
 "Flipkart's stand on net neutrality - The Hindu", http://www.thehindu.com/business/flipkarts-stand-on-net-neutrality/article7106072.ece
 "Our Internet is headed in the right direction: Amod Malviya - Livemint", http://www.livemint.com/Companies/1J4CaeGnXvKCbwvWW76J6H/Our-Internet-is-headed-in-the-right-direction-Amod-Malviya.html
© 2015, Abhinav Agarwal (अभिनव अग्रवाल). All rights reserved.
I went to the Maker Faire 2015 Bay Area show over the weekend. A lot of similarity to last year, but a few new things.
Sprout is an acquisition by HP, that they build a large touchpad and projector, as attachment to HP computer. It is kind of combination of projector, extended screen, touch screen, and working pad – that seems to blend physical things with virtual computer objects, such as capture objects into 3D graphics.
TechHive’s Mole-A-Whack is quite good station too – it is a reverse of classical Whack-A-Mole.
Here’s a video of it in action:
They use arduino-controlled Mole to whack kids who hide in the mole holes, but need raise head out of the hole cover (which is arduino-monitored), and reach to push a button (MaKey connected) to earn points.
The signals go into a Scratch program on computer for tally the winner.
This pipe organ is an impressive build:
As usual, lots of 3D printers, CNC mills, etc. and lots of drones flying.
Also I saw many college groups attending the events this year, bringing in all kinds of small builds for various applications.Possibly Related Posts:
- It’s Almost Maker Week
- AppsLab at the Maker Faire
- Maker Faire 2015
- Projecting Multiple Android Device Screens
- Raspi Shutdown Key
Recently, I had trouble starting ASM proxy instance on one of the nodes in my 2 node flex cluster having nodes host01 and host02. As a result I could not access the volume I created on an ASM diskgroup. This post explains how I resolved it.
While connected to host01, I created a volume VOL1 on DATA diskgroup with corresponding volume device /dev/asm/vol1-106 .
[grid@host01 root]$ asmcmd volcreate -G DATA -s 300m VOL1 [grid@host01 root]$ asmcmd volinfo -G DATA VOL1 Diskgroup Name: DATA Volume Name: VOL1 Volume Device: /dev/asm/vol1-106 State: ENABLED Size (MB): 320 Resize Unit (MB): 32 Redundancy: MIRROR Stripe Columns: 4 Stripe Width (K): 128 Usage: ACFS Mountpath:
I created ACFS file system on the newly created volume
[root@host01 ~]# mkfs -t acfs /dev/asm/vol1-106
I also created corresponding mount point /mnt/acfsmounts/acfs1 on both the nodes in the cluster.
root@host01 ~]# mkdir -p /mnt/acfsmounts/acfs1 root@host02 ~]# mkdir -p /mnt/acfsmounts/acfs1
When I tried to mount the volume device, I could mount the volume device on host01 but not on host02 .
[root@host01 ~]#mount -t acfs /dev/asm/vol1-106 /mnt/acfsmounts/acfs1 [root@host01 ~]# mount | grep vol1 /dev/asm/vol1-106 on /mnt/acfsmounts/acfs1 type acfs (rw) [root@host02 ~]# mount -t acfs /dev/asm/vol1-106 /mnt/acfsmounts/acfs1 mount.acfs: CLSU-00100: Operating System function: open64 failed with error data: 2 mount.acfs: CLSU-00101: Operating System error message: No such file or directory mount.acfs: CLSU-00103: error location: OOF_1 mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/vol1-106) mount.acfs: ACFS-02017: Failed to open volume /dev/asm/vol1-106. Verify the volume exists.
The corresponding volume device was visible on host01 but not on host02
[root@host01 ~]# cd /dev/asm [root@host01 asm]# ls vol1-106 [root@host02 ~]# cd /dev/asm [root@host02 asm]# ls
Since ADVM / ACFS utilize an ASM Proxy instance in a flex cluster to access metadata from a local / remote ASM instance , I checked whether ASM Proxy instance was running on both the nodes and realized that whereas ASM Proxy instance was running on host01, it was not running on host02
[root@host01 ~]# ps -elf | grep pmon | grep APX
0 S grid 27782 1 0 78 0 – 350502 – 10:09 ? 00:00:00 apx_pmon_+APX1
[root@host02 asm]# ps -elf | grep pmon | grep APX
[root@host01 ~]# srvctl status asm -proxy
ADVM proxy is running on node host01
[root@host01 ~]# crsctl stat res ora.proxy_advm -t
Name Target State Server State details
ONLINE ONLINE host01 STABLE
ONLINE OFFLINE host02 STABLE
I tried to start ASM proxy instance manually on host02
[grid@host02 ~]$ . oraenv ORACLE_SID = [grid] ? +APX2 The Oracle base has been set to /u01/app/grid [grid@host02 ~]$ sqlplus / as sysasm SQL*Plus: Release 22.214.171.124.0 Production on Sat May 2 10:31:45 2015 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORA-00099: warning: no parameter file specified for ASMPROXY instance ORA-00443: background process "VUBG" did not start SQL> ho oerr ORA 00443 00443, 00000, "background process \"%s\" did not start" // *Cause: The specified process did not start. // *Action: Ensure that the executable image is in the correct place with // the correct protections, and that there is enough memory.
I checked the memory allocated to VM for host02 – It was 1.5 GB as against 2.5 GB assigned to VM for host01. I increased the memory of host02 to 2.5 GB and ASM proxy instance started automatically.
[root@host01 ~]# crsctl stat res ora.proxy_advm -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.proxy_advm ONLINE ONLINE host01 STABLE ONLINE ONLINE host02 STABLE
Hope it helps!
Related Links :
Comments: 0 (Zero), Be the first to leave a reply!
You might be interested in this:
- 11g R2 RAC: Highly Available IP (HAIP)
- DB_FILE_MULTIBLOCK_READ_COUNT AND EXTENT SIZE
- CLONE AN EXISTING PDB AS NON-SYS USER
- MIGRATION OF A RAC DATABASE
- RECOVER VOTING DISK - SCENARIO-I
After my post Can you restore from a full online backup ?, I needed to come up with a report.
Assuming that each backup goes in a different directory, I just wrote two reports.
- Report gaps in v$backup_redolog (or rc_backup_redolog if you use the catalog)
DIR FIRST_CHANGE# NEXT_CHANGE# ------- ------------- ------------ /bck01/ 284891 285140 /bck01/ 285140 285178 /bck02/ 284891 285140 === GAP === /bck02/ 285178 285245 /bck03/ 285178 285245 /bck03/ 285245 286931 /bck03/ 286931 287803 /bck03/ 287803 288148
This could be done with analytics, by checking where the last next_change is not the current first_change, within a directory
SELECT dir, LAG missing_from_change#, first_change# missing_to_change# FROM ( SELECT REGEXP_REPLACE (handle, '[^/\]+$') dir, first_change#, next_change#, LAG(next_change#) OVER ( PARTITION BY REGEXP_REPLACE (handle, '[^/\]+$') ORDER BY first_change# ) LAG FROM v$backup_piece p JOIN v$backup_redolog l USING (set_stamp, set_count)) WHERE LAG != first_change#; DIR MISSING_FROM_CHANGE# MISSING_TO_CHANGE# ------- -------------------- ------------------ /bck02/ 285140 285178
- Reports directories where archivelogs don’t include changes (backup redolog) from the earliest to the latest checkpoint (backup datafile)
SELECT REGEXP_REPLACE (handle, '[^/\]+$') dir, MIN (checkpoint_change#), MAX (checkpoint_change#), MIN (first_change#), MAX (next_change#) FROM v$backup_piece p LEFT JOIN v$backup_datafile f USING (set_stamp, set_count) LEFT JOIN v$backup_redolog l USING (set_stamp, set_count) WHERE handle IS NOT NULL HAVING MIN (checkpoint_change#) < MIN (first_change#) OR MAX (checkpoint_change#) > MAX (next_change#) GROUP BY REGEXP_REPLACE (handle, '[^/\]+$'); DIR MINCHECKPT MAXCHECKPT MINFIRST MAXNEXT ------- ---------- ---------- ---------- ---------- /bck04/ 954292 954299 959487 1145473
the archives for the changes from 954292 to 959487 are missing.
If some archive backups are missing in one directory, it does not mean the database is irrecoverable, the archive backups could be in another directory. But it means that single directory would no longer permit you to restore or duplicate.
Another approach with RESTORE PREVIEW was provided by Franck in my previous post : List all RMAN backups that are needed to recover.
Usual disclaimer: there are plenty of other irrecoverabilty causes from hardware defect to backup “optimization” that are beyond the scope of this post.
May’s entry in the ‘How I Work’ series is PeopleSoft Blogger Iouri “Yury” Chadour. Yury has been sharing his knowledge on his Working Scripts blog for 7 years, so is a valuable and consistent member of our community. Yury’s site is full of tips, particularly new tools to try and techniques ‘around the edges’ of PeopleSoft. Thanks, and keep up the good work Yury!
Name: Iouri Chadour
Occupation: Vice President at Lazard Freres
Location: In the office in midtown NYC
Current computer: At work I use either standard Lenovo laptop or my VM client, my own machine is Lenovo X1 Carbon
Current mobile devices: Samsung Galaxy S3, iPad Air 2, Kindle Fire (Original)
I work: best when I have a set goal in mind – I like being able to check off my achievements from the list (more on that below.) As many others fellow bloggers have mentioned – challenge and ability to learn new things on the job are very important as well.
What apps/software/tools can’t you live without?
I use all of these Software Development Tools:
Notepad++ with lots of plugins PeopleCode user Defined language, Compare, AutoSave, NppExport, Explorer to name a few
Firefox with Firebug, AdBlock and Hootsuite
Feedly – this my main tool for following all the blogs and keeping up to date on the news
LastPass – very convenient password management for desktop and phone
KeePass – open source password manager
Toad for Oracle 12
Wunderlist – Android app and Desktop for Taks Management
Microsoft Project or Project Libre
Greenshot Screen Capture
Gimp – basic image editing
Besides your phone and computer, what gadget can’t you live without?
I like my original Kindle Fire – I use it for reading more than any other device.
What do you listen to while you work?
Listening really depends on the mood at time of the day. I mostly use Slacker Radio to listen to everything from contemporary and classic jazz, Classical to Parisian Electro and House music.
What PeopleSoft-related productivity apps do you use?
PeopleSoft Query Client for writing queries
Notepad++ to write and examine code and logs
TraceMagic for more advanced log review
Do you have a 2-line tip that some others might not know?
If I am stuck with a very difficult problem and can’t seem to find a good solution – I usually leave it and do something else – at some point the solution or a correct directions usually comes to my mind on it’s own.
What SQL/Code do you find yourself writing most often?
Since I work with a lot of Financials Modules so everything related to those modules. I do also write some tools related SQLs when I need to examine Process Scheduler tables.
What would be the one item you’d add to PeopleSoft if you could?
Code completion and Code/Project navigator – I use Notepad++ for now.
What everyday thing are you better at than anyone else?
I do not think I do something in particular better than anyone else, but I believe that I can be more efficient about some things than some of the people.
What’s the best advice you’ve ever received?
My family and my friends provided me with a lot of advice and support and I am greatly thankful for them being present in my life. But I do like the following quote:
“The more things that you read , the more things you will know. The more you learn, the more places you’ll go.” – Dr. Seuss
I talked with my clients at MemSQL about the release of MemSQL 4.0. Let’s start with the reminders:
- MemSQL started out as in-memory OTLP (OnLine Transaction Processing) DBMS …
- … but quickly positioned with “We also do ‘real-time’ analytic processing” …
- … and backed that up by adding a flash-based column store option …
- … before Gartner ever got around to popularizing the term HTAP (Hybrid Transaction and Analytic Processing).
- There’s also a JSON option.
The main new aspects of MemSQL 4.0 are:
- Geospatial indexing. This is for me the most interesting part.
- A new optimizer and, I suppose, query planner …
- … which in particular allow for serious distributed joins.
- Some rather parallel-sounding connectors to Spark. Hadoop and Amazon S3.
- Usual-suspect stuff including:
- More SQL coverage (I forgot to ask for details).
- Some added or enhanced administrative/tuning/whatever tools (again, I forgot to ask for details).
- Surely some general Bottleneck Whack-A-Mole.
There’s also a new free MemSQL “Community Edition”. MemSQL hopes you’ll experiment with this but not use it in production. And MemSQL pricing is now wholly based on RAM usage, so the column store is quasi-free from a licensing standpoint is as well.
Before MemSQL 4.0, distributed joins were restricted to the easy cases:
- Two tables are distributed (i.e. sharded) on the same key.
- One table is small enough to be broadcast to each node.
Now arbitrary tables can be joined, with data reshuffling as needed. Notes on MemSQL 4.0 joins include:
- Join algorithms are currently nested-loop and hash, and in “narrow cases” also merge.
- MemSQL fondly believes that its in-memory indexes work very well for nested-loop joins.
- The new optimizer is fully cost-based (but I didn’t get much clarity as to the cost estimators for JSON).
- MemSQL’s indexing scheme, skip lists, had histograms anyway, with the cutesy name skiplistogram.
- MemSQL’s queries have always been compiled, and of course have to be planned before compilation. However, there’s a little bit of plan flexibility built in based on the specific values queried for, aka “parameter-sensitive plans” or “run-time plan choosing”.
To understand the Spark/MemSQL connector, recall that MemSQL has “leaf” nodes, which store data, and “aggregator” nodes, which combine query results and ship them back to the requesting client. The Spark/MemSQL connector manages to skip the aggregation step, instead shipping data directly from the various MemSQL leaf nodes to a Spark cluster. In the other direction, a Spark RDD can be saved into MemSQL as a table. This is also somehow parallel, and can be configured either as a batch update or as an append; intermediate “conflict resolution” policies are possible as well.
In other connectivity notes:
- MemSQL’s idea of a lambda architecture involves a Kafka stream, with data likely being stored twice (in Hadoop and MemSQL).
- MemSQL likes and supports the Spark DataFrame API, and says financial trading firms are already using it.
Other application areas cited for streaming/lambda kinds of architectures are — you guessed it! — ad-tech and “anomaly detection”.
And now to the geospatial stuff. I thought I heard:
- A “point” is actually a square region less than 1 mm per side.
- There are on the order of 2^30 such points on the surface of the Earth.
Given that Earth’s surface area is a little over 500,000,000 square meters, I’d think 2^50 would be a better figure, but fortunately that discrepancy doesn’t matter to the rest of the discussion. (Edit: As per a comment below, that’s actually square kilometers, so unless I made further errors we’re up to the 2^70 range.)
Anyhow, if the two popular alternatives for geospatial indexing are R-trees or space-filling curves, MemSQL favors the latter. (One issue MemSQL sees with R-trees is concurrency.) Notes on space-filling curves start:
- In this context, a space-filling curve is a sequential numbering of points in a higher-dimensional space. (In MemSQL’s case, the dimension is two.)
- Hilbert curves seem to be in vogue, including at MemSQL.
- Nice properties of Hilbert space-filling curves include:
- Numbers near each other always correspond to points near each other.
- The converse is almost always true as well.*
- If you take a sequence of numbers that is simply the set of all possibilities with a particular prefix string, that will correspond to a square region. (The shorter the prefix, the larger the square.)
*You could say it’s true except in edge cases … but then you’d deserve to be punished.
Given all that, my understanding of the way MemSQL indexes geospatial stuff — specifically points and polygons — is:
- Points have numbers assigned to them by the space-filling curve; those are indexed in MemSQL’s usual way. (Skip lists.)
- A polygon is represented by its vertices. Take the longest prefix they share. That could be used to index them (you’d retrieve a square region that includes the polygon). But actually …
- … a polygon is covered by a union of such special square regions, and indexed accordingly, and I neglected to ask exactly how the covering set of squares was chosen.
As for company metrics — MemSQL cites >50 customers and >60 employees.