Mind the Gap: Bridging the Divide Between GenAI Promise and Practice
We are in a (yet another) hype data cycle. This time it’s GenAI. I get excited by it as much as any other data nerd. But more than that. As a businessman, I can see a raft of efficiencies GenAI can give companies in accelerating innovation, reducing costs from routine activities, and helping to streamline operations. Just think of the possibilities GenAI can give us! We are just a few steps away from J.A.R.V.I.S. No wonder the world is falling over to get on the GenAI bandwagon.
It has been a year since the general introduction of the GenAI. The hype keeps intensifying. The money poured into the field has never been so astronomical. The big tech companies and governments want to have their own versions. Yet, most businesses are taking their time implementing the solutions. Outside of the common applications like content creation, design, chatbots and co-pilots, they’re not yet making the full use of the technology. Why?
Experimental GenAI
What’s happening now is a mad scramble to put GenAI into use as soon as possible. Half of the business leadership see it as a source of competitive advantage. New products, lower staff cost, automation. While the other half fear being left behind. So, they both force speedy implementations, completely disregarding the state of their data, established processes, and culture.
The data teams are left with often the impossible tasks of bringing years of siloed and disparate data from all sorts of systems, storage and spreadsheets into training models. I mean, they weren’t able to figure out where the data sat up until now. What makes the business think that they can do it now (practically overnight)?
The implementation of GenAI is intricately linked to the challenges of the underlying enterprise data. There is no getting away from it. The best the data teams can do now is attempt to set up a training model with whatever data they can gather “on the surface”. A handful of reliable databases with core data (e.g. sales, customer, finance) that they know and trust. The rest is ad-hoc, depending on the ability to get hold of the additional sources, state of the data quality and even relevance. Snapshots at best.
Implementation complexity
Once the training is done, the data teams area told to put the model into production. And how are they supposed to do that (and continuously supplement it with new data) after just barely making it work in training?
That’s just the data side of things. You then face another can of worms to get the business users to adopt GenAI.
Many challenges affect the effectiveness with which organisations can leverage GenAI solutions. They’re not uniquely different from the business challenges normally associated with major initiatives that require changes to the BAU. The hype and the promise of quick riches are pushing the companies to ignore the basics of good change management.
Let’s see what’s involved.
Complexity of data and systems
GenAI models require high-quality, diverse, and well-integrated data to train effectively. The complexity of data sources and the difficulty of integrating them prove challenging in feeding Gen AI systems with the right data. This compromises their effectiveness and the accuracy of their outputs past initial training.
The dynamic nature of enterprise data, with its crazy mixture of formats and sources, requires production GenAI solutions that are adaptable and robust. The challenges come when you get to developing and maintaining these adaptable systems amidst complex data architectures. These take time and considerable effort to overcome.
Legacy systems and technical debt
Next, we have everyone’s favourite legacy systems. Integrating GenAI with entrenched legacy systems is a major PITA. These systems often lack the necessary interfaces or the flexibility to support advanced AI functionalities. Naturally, this leads to bottlenecks in deploying GenAI applications.
You then have years of the data “baggage” that has accumulated. Remember all those “can we just quickly” projects with no governance and bare hints of metadata in place? Well, it now comes to bite you. The technical debt makes it very difficult to implement new technologies without significant and costly refactoring, or even complete overhauls.
Organisational and cultural challenges
The introduction of GenAI technologies requires substantial change management efforts to align organisational culture, workflows, and employee roles with the new capabilities. Many businesses have not yet fully grasped the scale of the challenge.
This is probably the most terrifying aspect of GenAI to many employees, especially if they work in support roles that can be seen “at risk”. People don’t take kindly to the notion of potentially being replaced by a machine. Resistance to change can slow down or outright derail GenAI implementations.
If the company successfully quells staff insurrection, the deployment of GenAI also requires simultaneous expertise in AI/ML and domain-specific knowledge. To ensure that the AI systems are trained on relevant and contextually accurate data, you need business knowledge. It is not always possible to upskill the SMEs to learn AI (some can barely use Excel). Bridging this skill gap is a significant and costly hurdle.
Regulatory and compliance issues
GenAI systems often process vast amounts of personal and sensitive data, raising concerns about privacy, data protection, and compliance with regulations like GDPR or CCPA. Ensuring that GenAI applications comply with these regulations is another challenge that affects their deployment and operationalisation.
Beyond compliance, there is an increasing emphasis on ethical considerations and the responsible use of AI. Companies must navigate these considerations especially carefully to avoid reputational damage and legal challenges.
Cost and investment
Cost! My favourite aspect. The development, training, and deployment of GenAI systems carry significant costs, including computational resources, data storage, and expert personnel (lots of them). These costs can be prohibitive for many businesses, especially when considering the need for continuous data integration, system improvement and updating of AI models.
We often measure success via ROI. There is a significant uncertainty around the return on investment for GenAI projects. Especially those that are pioneering or experimental in nature, as many of them are at the moment. The uncertainty of the immediate tangible benefits makes it difficult to secure sufficient funding to scale GenAI.
Get the basics right
The challenges with enterprise data architecture stem from a complex interplay of technological, organisational, and financial factors. While the benefits of GenAI are promising, the path to adopting such technologies is not straightforward at all.
To fully leverage the potential of GenAI, companies must prioritise establishing a solid foundation in good data practices and organisational policies. This foundation involves ensuring high-quality, well-integrated data, addressing legacy system challenges, managing technical debt, fostering a culture of innovation and adaptability, and navigating regulatory and compliance issues effectively.
Good data practices begin with creating a unified, accessible data architecture that breaks down silos and integrates disparate data sources into a coherent system. You have kicked the can down the road long enough. The time of reckoning has finally arrived. You cannot enable efficient training and operation of GenAI models without accurate and relevant data. Companies must modernise systems, infrastructure, and data integration to support advanced AI functionalities without creating bottlenecks.
Finally, addressing cultural challenges is crucial. This involves implementing robust change management strategies to align organisational culture and workflows with GenAI capabilities. Companies must promote a culture of continuous learning and adaptability, encouraging employees to embrace new roles and opportunities that GenAI technologies bring. Remember, your employees are assets, not “deadwood”, unless your practices turn them into that.
Means don’t justify the end
I can’t stress enough the importance of managing the cost of investment in GenAI projects. Companies need to carefully plan their GenAI initiatives, balancing the initial costs against the potential for long-term efficiencies, innovation, and competitive advantage. This includes continuous investment in data integration, system improvement, and AI model updates.
You will have to address your technical debt and put in place proper data management and governance policies. GenAI will force you to adopt much more automation than you applied to date. It all adds to the total bill. Once you set off on the GenAI journey, it will push you to improve many aspects of your data and business practices.
If you don’t plan well, you run the risk your initial investment runs out as your GenAI gets bogged down. Budgets tend to underestimate the true costs of data projects by oversimplifying the challenges. But they will hit your wallet hard as you dip in the murky waters of your data estate.
Money and time, or lack of either, are the main roadblocks of all data projects. GenAI is no different. If it takes too long to deliver and costs more than the perceived benefits it brings, the management will kill it. Quicker than you think. It has happened to many tech advancements over the years that failed to deliver tangible value fast.
Control your costs with the utmost care.
Help to consider
This is where IOblend is a big help. IOblend could offer services and solutions designed to streamline the integration of GenAI technologies into business operations. And reduce the costs substantially:
Data integration solution that helps companies automate and organise their data for GenAI in a cost-effective manner.
Consulting services to navigate the complexities of legacy system integration and technical debt.
Change management support to address organisational and cultural challenges associated with the introduction of GenAI.
By partnering with IOblend, your business can effectively address the foundational needs for a successful GenAI implementation, paving the way for innovation, cost reduction, and enhanced operational efficiencies.
IOblend presents a ground-breaking approach to IoT and data integration, revolutionizing the way businesses handle their data. It’s an all-in-one data integration accelerator, boasting real-time, production-grade, managed Apache Spark™ data pipelines that can be set up in mere minutes. This facilitates a massive acceleration in data migration projects, whether from on-prem to cloud or between clouds, thanks to its low code/no code development and automated data management and governance.
IOblend also simplifies the integration of streaming and batch data through Kappa architecture, significantly boosting the efficiency of operational analytics and MLOps. Its system enables the robust and cost-effective delivery of both centralized and federated data architectures, with low latency and massively parallelized data processing, capable of handling over 10 million transactions per second. Additionally, IOblend integrates seamlessly with leading cloud services like Snowflake and Microsoft Azure, underscoring its versatility and broad applicability in various data environments.
At its core, IOblend is an end-to-end enterprise data integration solution built with DataOps capability. It stands out as a versatile ETL product for building and managing data estates with high-grade data flows. The platform powers operational analytics and AI initiatives, drastically reducing the costs and development efforts associated with data projects and data science ventures. It’s engineered to connect to any source, perform in-memory transformations of streaming and batch data, and direct the results to any destination with minimal effort.
IOblend’s use cases are diverse and impactful. It streams live data from factories to automated forecasting models and channels data from IoT sensors to real-time monitoring applications, enabling automated decision-making based on live inputs and historical statistics. Additionally, it handles the movement of production-grade streaming and batch data to and from cloud data warehouses and lakes, powers data exchanges, and feeds applications with data that adheres to complex business rules and governance policies.
The platform comprises two core components: the IOblend Designer and the IOblend Engine. The IOblend Designer is a desktop GUI used for designing, building, and testing data pipeline DAGs, producing metadata that describes the data pipelines. The IOblend Engine, the heart of the system, converts this metadata into Spark streaming jobs executed on any Spark cluster. Available in Developer and Enterprise suites, IOblend supports both local and remote engine operations, catering to a wide range of development and operational needs. It also facilitates collaborative development and pipeline versioning, making it a robust tool for modern data management and analytics
Unlock new capabilities with real time ACARS data
In this short article we are looking at one of the key data sources for the aviation industry – ACARS – and how IOblend helps to unlock new analytical capabilities from it.
Time to automate your airline’s DOC data
How to automate Direct Operating Cost (DOC) data collection, processing and serving with IOblend.
Automate airline fuel data collection & management
Collecting and managing airline fuel data is complex and time consuming. IOblend can greatly streamline the process and enable real-time decisioning.
The Data Mesh Gotchas!
I think most practitioners in the data world would agree that the core data mesh principles of decentralisation to improve data enablement are sound. Originally penned by Zhamak Dehghani, Data Mesh architecture is attracting a lot of attention, and rightly so.However, there is a growing concern in the data industry regarding how the data mesh
IOblend Data Mesh
IOblend Data Mesh – power to the data people! Analyst engineering made simple Hello folks, IOblend here. Hope you are all keeping well. Companies are increasingly leaning towards self-service data authoring. Why, you ask? It is because the prevailing monolithic data architecture (no matter how advanced) does not condone an easy way to manage the
Data lineage is a “must have”, not “nice to have”
Hello folks, IOblend here. Hope you are all keeping well. There is one thing that has been bugging us recently, which led to the writing of this blog. While working on several data projects with some of our clients, we observed instances when data lineage had not been implemented as part of the solutions. In