Welcome!

Recurring Revenue Authors: Flint Brenton, Elizabeth White, Yeshim Deniz, Xenia von Wedel, Liz McMillan

Blog Feed Post

Strata + Hadoop – Security in the Data Lake

Strata+hadoop world logoAbout two weeks ago I attended the Strata + HADOOP world conference, February 2015, in San Jose.  I think the most notable observation from an IT perspective is that people are getting serious about it. The second is that the technology driving the HADOOP ecosystem is evolving at warp speed.

ClickToTweet: Security in the Data Lake from Strata + Hadoop @AshvinKamaraju #DefenderOfData http://bit.ly/1FZjgkm

One example for security – Cloudera now says that “Security is Foundational” and the new default. Cloudera also said that they created a data governance partnership with Intel and Master Card. Items addressed included HDFS encryption zones, key management, Sentry unified authorization, data governance and authentication.  I’ve observed that the Cloudera distribution is addressing new security needs with every release, and this continues that trend. Another example – a session hosted by the security architect at VISA that I attended, where he definitely recommended using encryption in production deployments.

With this level of emphasis on security at the platform level, as well as additional application related controls that can be applied, such as field encryption or tokenization, the environment is making progress toward overcoming the security and compliance risks that customers must address in order to harness the value of all of the data from multiple sources.

The conference, organized by O-Reilly and Cloudera, has grown significantly in size this year and was hosted at the San Jose convention center for the first time. I was told that it was a sold out conference, and my additional observation was that all the sessions were definitely buzzing. The expo hall was huge and packed as well and I would say there were at least 100+ exhibitors. Microsoft and MapR were the primary sponsor, and attendees were primarily technical; technologists, engineers, data scientists, Big Data practitioners.  Most sessions were a match for this audience as they were technical in nature.

Walking away from this conference I would definitely say HADOOP and its ecosystem is “Big Data” and forms the Enterprise Data Lake. Everything else just fits or flows into this ecosystem. An informal poll in talking to the attendees leads me to believe that HADOOP is most definitely beginning to get deployed in production across virtually every industry segment – but most notably in the financial/banking, healthcare, retail and telco/media verticals.

The pace of innovation in the HADOOP ecosystem is happening at warp speed. Real time analytics were a very hot topic.  The hot item around this topic at this conference was a new compute engine called “Spark”, the hottest project at Apache and a potential replacement for Map Reduce. Databricks, a company in SF, is commercializing the technology and is perceived as the hottest new data company. The claim to fame for Spark is that it is a single compute engine that provides screamingly fast real time analytics and offers very simple APIs and programming paradigms (languages called Scala, R) that makes it very easy for application developers. A scientist from Databricks claimed that whereas time to sort 100tbytes with Map Reduce took 2000 machines about 72 minutes, SPARK could complete the sort using only207 machines in about 23 minutes. Predictions are that Map Reduce will be obsolete and Spark will be the new engine soon. Besides, SPARK, the other hot topics were machine learning and data visualization.

Of the HADOOP distributions, Cloudera and Hortonworks seem to have the most market share. However, MapR definitely seems to be garnering a lot of attention. They sponsored this conference, claimed they are the most deployed distribution in production, have many enterprise class features (replications, snapshots, mirroring etc.) etc. They also seem to be investing in a lot of Open Source Apache projects (Mesos, Drill etc.) that have a great deal of customer and community buy in.

It was also very heartening to get an update from DJ. Patil, the newly appointed Data Scientist for the Federal Government. Our fearless leader, President Obama, introduced DJ in a video clip, called Data Science a team sport and underscored the importance of Big Data.  I learned that a number of government data sets have been released to the public to spur innovation and research. DJ claimed Obama is the most data driven U.S. President, pointing to the executive order he’s issued to ensure that open and machine-readable data is the new government default policy.

Vormetric Secures Big Data

Vormetric Solutions Protect Data-at-rest Within Big Data Environments

Be sure to check out  data.gov which hosts 135000 data sets and executive order to ensure that open and machine-readable data is the new government default policy whitehouse.gov/usds (US Digital Service)

To wrap up, the pace of innovation and the value that the HADOOP ecosystem is bringing to the commercial world was definitely palpable. I  also asked some questions at a panel discussion of some data scientists/professional services folks from a small company called “Silicon Valley Data Scientists”. They said the adoption of HADOOP is real and happening across a number of industry segments. However, the “Data Lake” vision is not yet realized due to political and organizational boundaries. As scientists, they said the Data Lake will derive the maximum value from data (from all sources and from all the Lines of Businesses) and they are evangelizing the Enterprise Data Lake with all their clients. With the Data Lake comes the need for security, especially the need to zone the data and provide visibility on an as needed or authorized basis.

The post Strata + Hadoop – Security in the Data Lake appeared first on Data Security Blog | Vormetric.

Read the original blog entry...

More Stories By Vormetric Blog

Vormetric (@Vormetric) is the industry leader in data security solutions that span physical, big data and cloud environments. Data is the new currency and Vormetric helps over 1400 customers, including 17 of the Fortune 30 and many of the world’s most security conscious government organizations, to meet compliance requirements and protect what matters — their sensitive data — from both internal and external threats. The company’s scalable Vormetric Data Security Platform protects any file, any database and any application’s data —anywhere it resides — with a high performance, market-leading data security platform that incorporates application transparent encryption, privileged user access controls, automation and security intelligence.

@ThingsExpo Stories
DXWorldEXPO LLC, the producer of the world's most influential technology conferences and trade shows has announced the 22nd International CloudEXPO | DXWorldEXPO "Early Bird Registration" is now open. Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)
"We are a well-established player in the application life cycle management market and we also have a very strong version control product," stated Flint Brenton, CEO of CollabNet,, in this SYS-CON.tv interview at 18th Cloud Expo at the Javits Center in New York City, NY.
In his session at @ThingsExpo, Arvind Radhakrishnen discussed how IoT offers new business models in banking and financial services organizations with the capability to revolutionize products, payments, channels, business processes and asset management built on strong architectural foundation. The following topics were covered: How IoT stands to impact various business parameters including customer experience, cost and risk management within BFS organizations.
Here are the Top 20 Twitter Influencers of the month as determined by the Kcore algorithm, in a range of current topics of interest from #IoT to #DeepLearning. To run a real-time search of a given term in our website and see the current top influencers, click on the topic name. Among the top 20 IoT influencers, ThingsEXPO ranked #14 and CloudEXPO ranked #17.
While the focus and objectives of IoT initiatives are many and diverse, they all share a few common attributes, and one of those is the network. Commonly, that network includes the Internet, over which there isn't any real control for performance and availability. Or is there? The current state of the art for Big Data analytics, as applied to network telemetry, offers new opportunities for improving and assuring operational integrity. In his session at @ThingsExpo, Jim Frey, Vice President of S...
With the introduction of IoT and Smart Living in every aspect of our lives, one question has become relevant: What are the security implications? To answer this, first we have to look and explore the security models of the technologies that IoT is founded upon. In his session at @ThingsExpo, Nevi Kaja, a Research Engineer at Ford Motor Company, discussed some of the security challenges of the IoT infrastructure and related how these aspects impact Smart Living. The material was delivered interac...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
Amazon started as an online bookseller 20 years ago. Since then, it has evolved into a technology juggernaut that has disrupted multiple markets and industries and touches many aspects of our lives. It is a relentless technology and business model innovator driving disruption throughout numerous ecosystems. Amazon’s AWS revenues alone are approaching $16B a year making it one of the largest IT companies in the world. With dominant offerings in Cloud, IoT, eCommerce, Big Data, AI, Digital Assista...
The Founder of NostaLab and a member of the Google Health Advisory Board, John is a unique combination of strategic thinker, marketer and entrepreneur. His career was built on the "science of advertising" combining strategy, creativity and marketing for industry-leading results. Combined with his ability to communicate complicated scientific concepts in a way that consumers and scientists alike can appreciate, John is a sought-after speaker for conferences on the forefront of healthcare science,...
In his session at Cloud Expo, Alan Winters, U.S. Head of Business Development at MobiDev, presented a success story of an entrepreneur who has both suffered through and benefited from offshore development across multiple businesses: The smart choice, or how to select the right offshore development partner Warning signs, or how to minimize chances of making the wrong choice Collaboration, or how to establish the most effective work processes Budget control, or how to maximize project result...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
Personalization has long been the holy grail of marketing. Simply stated, communicate the most relevant offer to the right person and you will increase sales. To achieve this, you must understand the individual. Consequently, digital marketers developed many ways to gather and leverage customer information to deliver targeted experiences. In his session at @ThingsExpo, Lou Casal, Founder and Principal Consultant at Practicala, discussed how the Internet of Things (IoT) has accelerated our abilit...
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Two weeks ago (November 3-5), I attended the Cloud Expo Silicon Valley as a speaker, where I presented on the security and privacy due diligence requirements for cloud solutions. Cloud security is a topical issue for every CIO, CISO, and technology buyer. Decision-makers are always looking for insights on how to mitigate the security risks of implementing and using cloud solutions. Based on the presentation topics covered at the conference, as well as the general discussions heard between sessio...
Detecting internal user threats in the Big Data eco-system is challenging and cumbersome. Many organizations monitor internal usage of the Big Data eco-system using a set of alerts. This is not a scalable process given the increase in the number of alerts with the accelerating growth in data volume and user base. Organizations are increasingly leveraging machine learning to monitor only those data elements that are sensitive and critical, autonomously establish monitoring policies, and to detect...
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, discussed how they built...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settl...
IoT solutions exploit operational data generated by Internet-connected smart “things” for the purpose of gaining operational insight and producing “better outcomes” (for example, create new business models, eliminate unscheduled maintenance, etc.). The explosive proliferation of IoT solutions will result in an exponential growth in the volume of IoT data, precipitating significant Information Governance issues: who owns the IoT data, what are the rights/duties of IoT solutions adopters towards t...