I was pleased to attend Enterprise 2.0 Innovate on the West Coast for the first time. It occurred November 12 – 15 in the Santa Clara Convention Center. Here are my notes from this year’s Enterprise 2.0 2012 conference in Boston. Here are my notes from the session: Big Data: Everyone’s Challenge, speakers included Aditi Dhagat, Sr. Director, Adobe Information Management, Kaijun Zhan, IT Group Director and Senior Technologist, Cadence Design Systems, Randy Wagner, Drilling Advisor, Apache Corporation, and Johna Till Johnson, Analyst, Nemertes Research. Here is the session description.
“One of the biggest challenges with Big Data is the fact that it cuts across multiple domains. Everyone in the organization has a stake, from Sales and Marketing to Operations and IT. And even within IT, Big Data poses unique challenges for folks in infrastructure and Applications. How should companies organize to best ensure the success of their Big Data initiatives? This panel of experts discusses pros and cons of Big Data organizational and operational issues. Attendees will leave with an understanding of the challenges, pitfalls, and best practices of organizing a Big Data initiative.”
Johna started the session as the moderator. She said that most companies are not organized right to handle Big Data. This panel looked at this issue and all are doing Big Data work. Kaijun said that there is an IT manager responsible for Big Data and they hired some data scientists from a top university. Aditi said their Big Data effort is in its early mode. It is being driven by their company strategy. They are involved with digital marketing and this is one driver. Another is their suite of creative products. They are moving from a licensing model to a subscription company. They need to see how people are using their products. They want to move people from being free subscribers to paid subscribers.
Randy said his company is involved in oil drilling. They have been dealing with Big Data for along time using seismic data for oil exploration. He has started introducing social tools to the company. They are also looking new uses of Big Data for internal uses. Randy explained that seismic techniques are used to better find oil. They set off explosions and see how sound travels into the ground to understand what is there.
Johna asked about their initial thoughts as they put together their Big Data team. Kaijun said they were trying to do two things: promote business opportunities and increase effectiveness of the organization. One thing they looked at was help desk tickets to see what data was there that could be useful. They realized they needed to integrate the data scientists so these scientists understood their business model.
Randy asked about how many in the room work in IT and this was about 50% of the room. He said he is non-IT. In his case the business unit is leading the effort and went to the IT people to get it started. The business has been using a form of Big Data for years. Now they need real time data in drilling operations. Currently they spend about 20% of time in non-productive tasks in their drilling. Reducing this is huge cost reduction opportunity. Their long-term goal is to automate drilling operations. IT is a partner but not leading the charge.
Aditi said that Adobe is over 30 years old but still has a start up mentality. They are a software company and have been using Big Data for a long time. Their challenge is a bit different because so many parts of the organization are collecting data in different ways. This has led to some fragmented experiences for their customers. They need to connect all their sources of data. As a proof of concept they did this with aggregated HR data that included content from LinkedIn to predict who will be the future leaders of the company. This impressed senior leadership. There are also looking at what customers are saying about Adobe. They want to have a better understanding of their customers to micro-target potential customers.
It was asked how you determine what to data mine before you decide how to do it. Aditi said you need to determine the use case. For example, as Adobe changes to a subscription model they want to see how quickly they can move users from the free services to the paid services. Randy said it is not easy to determine what to data mine, as they do not know the unknowns. They are exploring their archive of data to see what is there before they focus.
Kaijun agreed that there is a learning curve in operation. He said that the available tools are expanding. It requires business skills and not just IT skills to properly do this work. Aditi said they are using four categories of tools: capture tools, tools to classify data, then data mining, and finally delivering and targeting data to business processes. There are also using visualization tools. Their own products come to play here.
Johna asked about how you identify data scientist. Aditi said they look for people interested in statistics, as well as people who do not know their business model so they have a fresh perspective. Kaijun agreed and said they look for people with a creative mind. Randy said that as a company they want to be contrarian and they look for people with this perspective.
It was asked about the need to speed. Randy that for them real time is a few seconds, not nanoseconds. Their data comes from the field through wireless so seconds is possible and they need to respond in minutes so this is okay. Johna asked about data integration of the different types of data. Aditi said that you need a rules engine that can look at the real time data and act as a filter. Randy said they are doing both old school and new school analysis on the same data.
Johna asked about the top three takeaways on Big Data. Randy said there will be many failures so anticipate this. Have your eyes wide open. Make sure you are addressing the right question. Do not skip steps. Aditi said think abut how you are going to connect all the different types of data in a common model. There will be a bit of re-skilling people to build data models. Find a good business unit to partner with. In her case, it is product marketing. Kaijun said that you need to produce fast as people will lose patience.