A Documentation Infrastructure For The Management Of Data Mining Projects Download
Data management is the procedure of ingesting, storing, organizing and maintaining the data created and collected by an organisation. Constructive information management is a crucial piece of deploying the Information technology systems that run business applications and provide analytical information to help drive operational determination-making and strategic planning by corporate executives, business concern managers and other finish users.
The data management process includes a combination of different functions that collectively aim to make sure that the data in corporate systems is accurate, available and attainable. Most of the required work is done by IT and data management teams, but business users typically also participate in some parts of the procedure to ensure that the data meets their needs and to get them on board with policies governing its utilize.
This comprehensive guide to information management further explains what information technology is and provides insight on the individual disciplines it includes, best practices for managing information, challenges that organizations face up and the business benefits of a successful information management strategy. Y'all'll too find an overview of data management tools and techniques. Click through the hyperlinks on the page to read nearly data management trends and get expert communication on managing corporate data.
Importance of data management
Data increasingly is seen as a corporate asset that tin be used to make more-informed business concern decisions, meliorate marketing campaigns, optimize business operations and reduce costs, all with the goal of increasing revenue and profits. Only a lack of proper information management can saddle organizations with incompatible data silos, inconsistent data sets and data quality problems that limit their ability to run business concern intelligence (BI) and analytics applications -- or, worse, lead to faulty findings.
Data management has also grown in importance every bit businesses are subjected to an increasing number of regulatory compliance requirements, including data privacy and protection laws such as GDPR and the California Consumer Privacy Act. In add-on, companies are capturing e'er-larger volumes of information and a wider variety of data types, both hallmarks of the large information systems many take deployed. Without adept information management, such environments tin can become unwieldy and hard to navigate.
Types of data management functions
The separate disciplines that are part of the overall data management process cover a serial of steps, from data processing and storage to governance of how information is formatted and used in operational and analytical systems. Evolution of a data compages is often the first step, peculiarly in large organizations with lots of data to manage. An compages provides a pattern for the databases and other data platforms that will be deployed, including specific technologies to fit individual applications.
Databases are the nearly mutual platform used to concur corporate data; they contain a collection of data that'due south organized so it tin can be accessed, updated and managed. They're used in both transaction processing systems that create operational information, such as customer records and sales orders, and data warehouses, which store consolidated data sets from business systems for BI and analytics.
Database administration is a cadre information management part. Once databases have been set up, functioning monitoring and tuning must be done to maintain adequate response times on database queries that users run to get data from the data stored in them. Other administrative tasks include database design, configuration, installation and updates; information security; database backup and recovery; and awarding of software upgrades and security patches.
The principal engineering science used to deploy and administer databases is a database management organization (DBMS), which is software that acts as an interface between the databases it controls and the database administrators, end users and applications that access them. Culling data platforms to databases include file systems and cloud object storage services; they shop information in less structured ways than mainstream databases do, which offers more flexibility on the types of information that tin exist stored and how it'due south formatted. Equally a result, though, they aren't a proficient fit for transactional applications.
Other cardinal information management disciplines include data modeling, which diagrams the relationships between data elements and how information flows through systems; data integration, which combines information from different information sources for operational and belittling uses; data governance, which sets policies and procedures to ensure information is consistent throughout an organization; and data quality management, which aims to set up data errors and inconsistencies. Another is master data management (MDM), which creates a common set of reference information on things like customers and products.
Data direction tools and techniques
A wide range of technologies, tools and techniques can be employed as part of the data management process. That includes the following available options for different aspects of managing data.
Database management systems. The most prevalent type of DBMS is the relational database management system. Relational databases organize data into tables with rows and columns that contain database records; related records in different tables tin be connected through the use of primary and foreign keys, fugitive the need to create duplicate data entries. Relational databases are built effectually the SQL programming language and a rigid data model best suited to structured transaction data. That and their support for the Acid transaction properties -- atomicity, consistency, isolation and immovability -- take fabricated them the top database choice for transaction processing applications.
All the same, other types of DBMS technologies have emerged every bit feasible options for dissimilar kinds of data workloads. Most are categorized equally NoSQL databases, which don't impose rigid requirements on data models and database schemas; as a result, they can store unstructured and semistructured information, such as sensor information, cyberspace clickstream records and network, server and awarding logs.
There are iv main types of NoSQL systems: certificate databases that shop data elements in document-similar structures, key-value databases that pair unique keys and associated values, broad column stores with tables that have a large number of columns, and graph databases that connect related data elements in a graph format. The NoSQL name has become something of a misnomer -- while NoSQL databases don't rely on SQL, many now support elements of it and offering some level of ACID compliance.
Additional database and DBMS options include in-memory databases that shop data in a server'south memory instead of on deejay to accelerate I/O performance and columnar databases that are geared to analytics applications. Hierarchical databases that run on mainframes and predate the evolution of relational and NoSQL systems are also still available for use. Users can deploy databases in on-premises or cloud-based systems; in addition, various database vendors offer managed cloud database services, in which they handle database deployment, configuration and administration for users.
Big information management. NoSQL databases are oftentimes used in big data deployments considering of their power to store and manage various data types. Big data environments are also commonly congenital effectually open source technologies such equally Hadoop, a distributed processing framework with a file organization that runs across clusters of commodity servers; its associated HBase database; the Spark processing engine; and the Kafka, Flink and Tempest stream processing platforms. Increasingly, large data systems are beingness deployed in the cloud, using object storage such as Amazon Simple Storage Service (S3).
Data warehouses and information lakes. Two alternative repositories for managing analytics information are data warehouses and data lakes. Data warehousing is the more traditional method -- a data warehouse typically is based on a relational or columnar database, and information technology stores structured data pulled together from different operational systems and prepared for assay. The principal information warehouse use cases are BI querying and enterprise reporting, which enable business analysts and executives to analyze sales, inventory direction and other key performance indicators.
An enterprise information warehouse includes data from business systems across an organization. In large companies, private subsidiaries and business concern units with direction autonomy may build their own information warehouses. Data marts are some other choice -- they're smaller versions of data warehouses that contain subsets of an organization's data for specific departments or groups of users.
Data lakes, on the other hand, shop pools of large data for employ in predictive modeling, motorcar learning and other avant-garde analytics applications. They're most commonly built on Hadoop clusters, although data lake deployments are also done on NoSQL databases or deject object storage; in improver, dissimilar platforms can be combined in a distributed data lake environment. The data may be candy for analysis when information technology's ingested, but a information lake often contains raw data stored equally is. In that instance, data scientists and other analysts typically practice their own data preparation piece of work for specific analytical uses.
Data integration. The well-nigh widely used data integration technique is excerpt, transform and load (ETL), which pulls information from source systems, converts it into a consequent format and then loads the integrated data into a data warehouse or other target system. Still, information integration platforms now also support a variety of other integration methods. That includes extract, load and transform (ELT), a variation on ETL that leaves information in its original form when information technology'southward loaded into the target platform. ELT is a common option for data integration jobs in information lakes and other big information systems.
ETL and ELT are batch integration processes that run at scheduled intervals. Information direction teams tin as well do existent-time data integration, using methods such every bit change data capture, which applies changes to the data in databases to a data warehouse or other repository, and streaming data integration, which integrates streams of real-fourth dimension information on a continuous basis. Data virtualization is some other integration option -- it uses an abstraction layer to create a virtual view of information from unlike systems for terminate users instead of physically loading the data into a data warehouse.
Information governance, information quality and MDM. Data governance is primarily an organizational procedure; software products that can assistance manage data governance programs are available, but they're an optional chemical element. While governance programs may exist managed by data direction professionals, they commonly include a information governance quango made upwards of business executives who collectively make decisions on common data definitions and corporate standards for creating, formatting and using data.
Another key aspect of governance initiatives is information stewardship, which involves overseeing data sets and ensuring that end users comply with the approved information policies. Data steward tin can be either a full- or role-time position, depending on the size of an system and the telescopic of its governance program. Data stewards can likewise come from both business operations and the Information technology department; either way, a shut noesis of the data they oversee is normally a prerequisite.
Data governance is closely associated with information quality improvement efforts; metrics that document improvements in the quality of an organization'south data are fundamental to demonstrating the business value of governance programs. Data quality techniques include data profiling, which scans data sets to identify outlier values that might be errors; information cleansing, as well known every bit data scrubbing, which fixes data errors by modifying or deleting bad data; and data validation, which checks data against preset quality rules.
Main data management is besides affiliated with information governance and data quality, although MDM hasn't been adopted as widely as the other two data direction functions. That's partly due to the complexity of MDM programs, which mostly limits them to large organizations. MDM creates a central registry of master data for selected data domains -- what'southward often called a golden record. The master information is stored in an MDM hub, which feeds the data to analytical systems for consequent enterprise reporting and assay; if desired, the hub tin can also push updated master information back to source systems.
Data modeling. Data modelers create a serial of conceptual, logical and physical data models that document data sets and workflows in a visual form and map them to business requirements for transaction processing and analytics. Common techniques for modeling data include the development of entity human relationship diagrams, data mappings and schemas. In add-on, information models must be updated when new data sources are added or an organization's information needs changes.
Data management best practices
A well-designed data governance program is a disquisitional component of effective information management strategies, especially in organizations with distributed data environments that include a diverse fix of systems. A strong focus on data quality is too a must. In both cases, though, IT and information management teams can't go it alone. Concern executives and users take to be involved to make sure their information needs are met and data quality bug aren't perpetuated. The same applies to data modeling projects.
Too, the multitude of databases and other information platforms available to be deployed requires a careful approach when designing a data architecture and evaluating and selecting technologies. It and information managers must exist sure the systems they implement are fit for the intended purpose and will evangelize the data processing capabilities and analytics data required past an arrangement's business organization operations.
DAMA International, the Data Governance Professionals System and other industry groups work to accelerate understanding of data management disciplines and offer best-practices guidance. For example, DAMA has published DAMA-DMBOK: Data Direction Body of Noesis, a reference volume that attempts to define a standard view of information direction functions and methods. Commonly referred to equally the DMBOK, the book was first published in 2009; a DMBOK2 second edition was released in 2017.
Data management risks and challenges
If an organization doesn't have a well-designed data architecture, it can terminate up with siloed systems that are hard to integrate and manage in a coordinated fashion. Fifty-fifty in better-planned environments, enabling data scientists and other analysts to find and access relevant information can be a challenge, specially when the information is spread across various databases and big information systems. To assist make data more accessible, many information direction teams are creating data catalogs that document what's available in systems and typically include business organization glossaries, metadata-driven data dictionaries and information lineage records.
The shift to the cloud can ease some aspects of information management work, but it also creates new challenges. For instance, migrating to cloud databases and large data platforms can be complicated for organizations that need to move data and processing workloads from existing on-premises systems. Costs are some other big upshot in the cloud -- the apply of cloud systems and managed services must be monitored closely to make certain data processing bills don't exceed the approaching amounts.
Many data direction teams are now among the employees who are accountable for protecting corporate data security and limiting potential legal liabilities for data breaches or misuse of data. Data managers demand to aid ensure compliance with both government and manufacture regulations on data security, privacy and usage. That has go a more pressing business concern with the passage of GDPR, the European Union's data privacy law that took outcome in May 2018, and the California Consumer Privacy Act, which was signed into law in 2018 and is scheduled to go constructive at the first of 2020.
Data direction tasks and roles
The data management process involves a wide range of tasks, duties and skills. In smaller organizations with express resources, individual workers may handle multiple roles. Only in full general, information direction professionals include data architects, data modelers, database administrators (DBAs), database developers, data quality analysts and engineers, data integration developers, data governance managers, data stewards and data engineers, who work with analytics teams to build information pipelines and prepare data for assay.
Information scientists and other information analysts may as well handle some information management tasks themselves, especially in big data systems with raw data that needs to be filtered and prepared for specific uses. Likewise, awarding developers often help deploy and manage big data environments, which require new skills overall compared to relational database systems. As a consequence, organizations may take to hire new workers or retrain traditional DBAs to meet their large data management needs.
Benefits of expert data management
A well-executed data management strategy can help companies gain potential competitive advantages over their business rivals, both by improving operational effectiveness and enabling amend decision-making. Organizations with well-managed data can as well become more than agile, making it possible to spot marketplace trends and move to take reward of new business opportunities more than chop-chop.
Effective data management tin also help companies avoid data breaches, information privacy bug and regulatory compliance issues that could impairment their reputation, add together unexpected costs and put them in legal jeopardy. Ultimately, the biggest do good that a solid approach to data management tin provide is amend business performance.
Data management history and evolution
The first flowering of information management was largely driven by It professionals who focused on solving the problem of garbage in, garbage out in the earliest computers afterwards recognizing that the machines reached false conclusions considering they were fed inaccurate or inadequate data.
Get-go in the 1960s, industry groups and professional associations promoted best practices for data direction, especially in terms of professional training and data quality metrics. Mainframe-based hierarchical databases also became available that decade.
The relational database emerged in the 1970s then cemented its place at the middle of the data management procedure in the 1980s. The idea of the data warehouse was conceived in the belatedly 1980s, and early adopters of the concept began deploying data warehouses in the mid-1990s. By the early on 2000s, relational software was a dominant technology, with a virtual lock on database deployments.
Simply the initial release of Hadoop became available in 2006 and was followed by the Spark processing engine and various other big information technologies. A range of NoSQL databases also started to go available in the same fourth dimension frame. While relational technology still has the largest share past far, the rise of large data and NoSQL alternatives and the new data lake environments they enable has given organizations a broader set of information management choices.
This was last updated in Oct 2019
Keep Reading Almost What is data management and why is it important?
- Data management advice from the Pentagon's chief information officer
- Information governance and a good data architecture go hand in hand
- Tips on managing data quality improvement projects
- How companies are dealing with GDPR's rules on managing data
- The role of effective data management in the coming data apocalypse
DOWNLOAD HERE
Posted by: martellonurs1937.blogspot.com
0 Komentar
إرسال تعليق