The Biggest Data Engineering Challenges of 2022
The dynamic data engineering technology space of today is propelled ahead by the decisive shift from on-premise databases and BI tools to modern and advanced cloud-based data platforms built on lakehouse architecture. Today’s challenging data environment stipulates, reliance on multiple technologies to keep up with scale, speed and use cases. The cloud data technology market is advancing swiftly, and includes an extensive set of open source and commercial data technologies, tools, and products, unlike the on-premise data warehouses of the past. Organisations are developing DataOps Frameworks and Functions to be maximize the value of their data and to be in relevance. To enable automated and continuous delivery of data to business intelligence analytics and data powered products, processes and DataOps tools are in place. As per a recent study by Gradient Flow and Immuta, respondents cited these areas as the most challenging in the data engineering space: Data Quality and Validation; Monitoring and Auditing for Compliance; Masking and Anonymization; and Data Discovery. We at Ignitho have been building capabilities, across our Data Engineering Practice to mitigate these challenges in the most holistic way possible. With potential sources of error increasing day by day; factors like volume, variety, velocity, data source type, and the number of data providers, are playing a huge role in how we face the concern of data quality. Solutions based on our Informatica and TensorFlow Data Validation capabilities have helped our clients tackle data quality issues and challenges impacting critical and products, depending on the accuracy of Data. At times, when organizations’ have to handle myriad of data sources and data types, such as unstructured data consisting of text, images, audio, and video; the data integration solutions with Apache Spark, dbt and Hive as well as managed services like AWS Glue, Dataform and Azure Data Factory has been adding value to our clientele’s data compliance efforts. Data Masking and Data Encryption are distinct data privacy solutions. Ignitho’s regulated approach on GDPR, HIPAA, CCPA, and SOC 2 has been pivotal in removing the misconception among data stewards on Data Governance and Data Privacy when considering data anonymization solutions, that encrypted data is indeed a form of data masking. A recent study indicated that close to 1 in every 4 organizations, do not have a structured Data Catalog or Discovery tool. As they grow, the amount of raw and derived data they generate and store rises, and users need tools to help them discover the right data resources. Ignitho’s expertise in the areas of Google Data Catalog, Collibra, and Azure Data Catalog, are assisting organizations drive adoptability. Headquartered in the US, Ignitho services its customers through its offices in London, UK, and New York and Richmond, USA, and its development centers in Brighton, UK and Kochi, India. We are rapidly gaining recognition as a market leader in Frugal Technology Innovation for Enterprises – the ability to do more with less.
Introducing the Frugal approach to Technology Innovation in Enterprises
In the following 3-part blog series, the aim is to impart a basic knowledge of what “frugal innovation” actually stands for and what it means in an enterprises’ context. The next two blogs will talk about the principles that guide frugal innovators and why enterprise CIOs must take a closer look at the possibilities of frugal innovation. “Frugal Innovation” is defined as the art of overcoming harsh constraints and innovating sustainable products with limited resources. As a concept, it is not new and has been around for at least a decade, largely prevailing in the socio-economic context of developing countries like India, Kenya, Brazil, and China, where we have seen examples of frugal innovation by grassroots entrepreneurs to multinationals across sectors. At the grassroots end, examples include clay refrigerators used to store vegetables in rural areas where the lack of electricity is a constraint to baby warmers designed to help premature babies maintain their body temperature and preserve their lives. At the multinational end, pioneers of frugal innovation include the likes of GE, Siemens and Renault-Nissan, who have built low-cost solutions overcoming the constraints of energy and environmental challenges. Frugal Innovation creates value for Individuals, Businesses & Society Frugal Innovation begins with identifying opportunities that arise from adversity. Innovative minds reframe problems and convert them into opportunities. Maximum effort is made to flex available assets and convert them into sustainable solutions appropriate to the problem, the situation and the end user. Frugal innovation, therefore, unleashes the creativity in individuals and helps them to do more and better with less. Frugal innovation is thus also geared towards creating value for individuals, business, and society through the efficient use of available resources. Because of the transformative power of the concept, I have been researching it along with co-authors for over a decade now. In 2012, along with Navi Radjou, an innovation guru and winner of the Thinkers50 Innovation Award, and Simone Ahuja, a film-maker and consultant, I wrote Jugaad Innovation, a bestselling book about frugal innovation in emerging markets and what large Western corporations can learn from their emerging market counterparts. Hailed by the Economist as ‘the most comprehensive book yet to appear on the subject of Frugal Innovation’, the concept of frugal innovation attracted a lot of attention in the West for the West. So, Navi Radjou and I then went on to publish a follow-up book called Frugal Innovation: How to Do More with Less that focused specifically on frugal innovation in developed economies in the 21st century. In the book, we look at large companies such as Unilever, Renault, GE, Siemens, GSK, and others that are using the frugal principles to drive innovation in resource-constraint environments globally. Frugal Technology Innovation for Enterprises Over the last decade or so, we have seen a tremendous explosion of frugal innovation in various socio-economic contexts. Now an interesting question arises: can frugal innovation be applied to bring technology innovation to enterprises? Ignitho has taken the bold, pioneering step of adapting this concept to drive Frugal Technology Innovation for Enterprises. I have been collaborating closely with Joseph Olassa, co-founder and CEO of Ignitho, to actively promote Frugal Technology Innovation for Enterprises in the US and UK markets. Joseph has been a leading advocate of this approach and has presented his ideas at various global conferences. Combining my research on frugal innovation with Joseph’s expertise in technology innovation, Ignitho and I have successfully developed a frugal methodology to rapidly prototype business ideas and scale technology innovation in enterprises facing time and resource constraints. Ignitho’s disruptive new approach can help unlock efficiencies to innovate using available resources (money, time, people), to deliver tangible outcomes for the business using technology. If the concept of Frugal Technology Innovation interests you, stay tuned for my next blog which talks about the principles guiding Frugal Innovators and how they can lead enterprises to achieve successful Technology Innovation.
4 Key Principles For CIO While Selecting An Innovation Partner
Introduction: In today’s complex and fast changing world, the role of the CIO is not an easy one. Balancing the demands of the commercial business, key IT sponsored programmes, compliance (The California Online Privacy Protection Act, The Computer Security Act of 1997 and GDPR to name a few) and let us not even go into the territory of Digital and Agile working practices! It’s not surprising that IT innovation usually ends up low down the priority list. Add this to the challenge of securing funding for efforts that may not have clear paybacks then the result is Technology Innovation is often ignored completely! Is this acceptable? Technology Innovation – Ownership Well the short answer would be everyone and anyone in an enterprise could and should have a role. But where should ownership reside? I set it hard to argue that there is no better place than that of the office of the CIO. The question CIO’s need to ask is ‘am I comfortable’ with that responsibility and maybe ‘am I comfortable’ with it outside IT? If you come to the conclusion that you should own it then you need to take actions to make this happen. Plan In order to really gain momentum, there are a few key principles that need to be considered: The need to allow anybody in an enterprise to put forward ideas – Technology ideas are no longer the exclusive domain of IT. Accept and embrace it. The enterprise innovation groups are bombarded with unlimited ideas coming from various internal teams. The issue is in identifying the right idea and the required time to nurture these ideas. Keep the process light – don’t hide behind monolithic processes (read barriers) – The CIO’s we speak to often say that they mostly qualify ideas based on factors such as exertion, money, production, and achievement. This process is high cost and time-consuming, resulting in ideas being scrapped even before they are properly considered. The key is to identify ideas with minimum effort and filter the ones with maximum potential benefit. This is easier said than done using monolithic processes in innovation. Resources – plan some time to look at ideas to find the best potential innovation idea. While considering an idea CIOs are forced to consider the following factors such as the effort, capital and output required, to turn these ideas to functional products. Budget – allocate some resource/money – In today’s digital era, CIO’s often shy away from adopting a ‘silver bullet’ approach to technology innovation. CIO’s should think of ways of starting small and gain some traction. Be smart – look into the market for ideas – for example have you looked at RPA and how you could use it to improve your business processes (including IT) or extend a legacy system or two. What are businesses in other industries doing? Could you ‘migrate’ the model to your business? Partner – To build a competitive advantage, enterprises need to carefully select their innovation partners. Enterprises should look to select an innovation partner who is able to bring a global delivery model leveraging onsite and offshore advantages to the party. Innovation Partner For those ideas that require software to be built the golden rule is not to spend any more than you must in order to prove or disprove the idea. Look for a partner who does not treat innovation projects the same way as mainstream IT projects – business cases/project and expects the same guarantees on the investment return. Enterprises should look for innovation partners who give preference to ROI-backed innovation or in other terms sustainable innovation. An equally cost-effective method for sustainable innovation is adoption of Frugal Innovation methodology for technology innovation by enterprises. Ignitho Technologies, headquartered in the USA, and with its offices in New York, USA & London, UK, and its innovation labs & development centres in Richmond, USA, Brighton, UK and Kochi, India, specialises in Digital Applications – The ability to start small and deliver quick outcomes which are inherently scalable, for the web, mobile and cloud and Innovation Pods – The ability to build focused agile pods with cross-functional teams delivering faster prototypes and digital solutions. The secret of success for Ignitho lies in their Frugal Innovation methodology – “the ability to do more with less” – developed in collaboration with a world-renowned thought leader from the University of Cambridge. With its Frugal Technology Innovation methodology, Ignitho works with enterprises to identify potential ideas and using the geographically spread innovation labs to build MVPs. This allows enterprises to take up a step-by-step approach to technology innovation instead of going for the big bang approach which could, in turn, hurt their IT budget. In summary, the CIO’s should invest the time and take the lead on Innovation and should not be afraid to do so. There are great tools and partners who can help CIO’s succeed on the journey. You may not find a ‘silver bullet’ idea but if you start the journey with a partner like Ignitho, you will find things that can improve your business and that can’t be a bad thing.
A CIO’s guide to the need for Frugal Technology Innovation in Enterprises
In our previous two blogs in this series, we discussed what Frugal Innovation is, and the five principles that guide Frugal Innovators. While still relatively unknown in an enterprise context, the Frugal approach to Technology Innovation in the Enterprise may just be the golden ticket for CIO’s and business leaders to help escalate the pace on effective innovation at a time of rapid business disruptions caused by COVID-19. CIO’s and business leaders already recognize that innovation is no longer a luxury but a necessity for an enterprise today. Recent corporate history shows that innovation could well be the difference between exponential success or rapid decline in the enterprise. Consider the well-known examples of Kodak and Blockbuster, giants in their time but who no longer exist today out of poor reactions to the changing business environment and consumer behavior. Kodak underestimated the potential of digital photography which later disrupted the entire industry and replaced its film-based photography. Similarly, when Blockbuster CEO John Antioco and his team laughed at the proposal of partnership with Netflix in the year 2000, little did they know what waited for them in the coming years. In an enterprise context, innovation is normally a result of a burning need, an emerging trend or a popular new technology platform, or a convergence of these. For example, look at how the enterprise landscape has changed because of the coupling of a need with a new technology trend, such as gaming, social media and the emergence of super-powerful smartphones and tablets. When mobile took over the user experience factor, businesses had to adapt and deliver mobile-friendly applications to attract and retain their customers. While clearly recognizing the need to innovate quickly, enterprise CIOs face practical challenges in using a one-size-fits-all, big bang approach to all technology innovation. Our discussions with over 100 CIO’s have thrown up the following top issues. Lots of ideas but no sufficient bandwidth to nurture Innovation in an enterprise is often not a problem of finding ideas. In many scenarios, the CIO or innovation group is bombarded with a plethora of ideas coming from various internal sources. The problem therefore really lies in finding the necessary bandwidth to nurture these ideas, to run alongside larger transformation initiatives and business-as-usual. Need help in qualifying ideas and creating business cases The CIO’s we spoke to tell us that they would welcome advice and extended bandwidth to qualify ideas based on factors such as effort, capital, output, and success. Often though, this comes at a high financial cost and may also be time-consuming, resulting in ideas either being dropped or a loss of the window of opportunity. Identifying ideas that require minimum effort and provide maximum output is easier said than done using conventional approaches to innovation. Not enough good ideas that qualify While there may be a long list of potential innovation ideas, sometimes innovators face the issue of good quality ideas. When the success rates of these available ideas are compared with certain metrics such as the effort, capital and output required, many of them fall off the scale, resulting in a lot fewer ideas. Limited budgets can only nurture a few ideas Big bang transformational approaches to innovation are normally very expensive and time-intensive, thereby consuming whatever little budgets were available in the first place. Once again, this may cause other potentially brilliant ideas to fall by the wayside for lack of available budget and resources. No designated budget for innovation in non-core solutions Surprising as it may sound in today’s digital era, CIO’s are still often stifled by the lack of appetite within the enterprise to invest in new ideas. They must work very hard to push an agenda of innovation to run alongside business as usual initiatives. As a result, most ideas fail to get off the ground using the traditional big bang approach to innovation. Frugal Technology Innovation may be the answer Ignitho’s Frugal Technology Innovation methodology (Doing More with Less), built-in conjunction with Jaideep Prabhu, one of the world’s leading authorities and best-selling author on the subject, helps tangibly demonstrate ideas to the business stakeholders using limited resources through Rapid Prototyping, which can be ramped up to Scalable Solutions based on early success. Ignitho’s Innovation Labs, its unique peer ecosystem, and proven high-quality business and technical resources, are already translating business ideas into successful reality for enterprises. Talk to us today to find out more and get started on your own Frugal Technology Innovation journey in your Enterprise.