Talend Tutorial - Add Agility To Data

Talend Tutorial - Add Agility To Data Talend Tutorial Future Of Data Integration Back Home Categories Online Courses Mock Interviews Webinars NEW Community Write for Us Categories Artificial Intelligence AI vs Machine Learning vs Deep LearningMachine Learning AlgorithmsArtificial Intelligence TutorialWhat is Deep LearningDeep Learning TutorialInstall TensorFlowDeep Learning with PythonBackpropagationTensorFlow TutorialConvolutional Neural Network TutorialVIEW ALL BI and Visualization What is TableauTableau TutorialTableau Interview QuestionsWhat is InformaticaInformatica Interview QuestionsPower BI TutorialPower BI Interview QuestionsOLTP vs OLAPQlikView TutorialAdvanced Excel Formulas TutorialVIEW ALL Big Data What is HadoopHadoop ArchitectureHadoop TutorialHadoop Interview QuestionsHadoop EcosystemData Science vs Big Data vs Data AnalyticsWhat is Big DataMapReduce TutorialPig TutorialSpark TutorialSpark Interview QuestionsBig Data TutorialHive TutorialVIEW ALL Blockchain Blockchain TutorialWhat is BlockchainHyperledger FabricWhat Is EthereumEthereum TutorialB lockchain ApplicationsSolidity TutorialBlockchain ProgrammingHow Blockchain WorksVIEW ALL Cloud Computing What is AWSAWS TutorialAWS CertificationAzure Interview QuestionsAzure TutorialWhat Is Cloud ComputingWhat Is SalesforceIoT TutorialSalesforce TutorialSalesforce Interview QuestionsVIEW ALL Cyber Security Cloud SecurityWhat is CryptographyNmap TutorialSQL Injection AttacksHow To Install Kali LinuxHow to become an Ethical Hacker?Footprinting in Ethical HackingNetwork Scanning for Ethical HackingARP SpoofingApplication SecurityVIEW ALL Data Science Python Pandas TutorialWhat is Machine LearningMachine Learning TutorialMachine Learning ProjectsMachine Learning Interview QuestionsWhat Is Data ScienceSAS TutorialR TutorialData Science ProjectsHow to become a data scientistData Science Interview QuestionsData Scientist SalaryVIEW ALL Data Warehousing and ETL What is Data WarehouseDimension Table in Data WarehousingData Warehousing Interview QuestionsData warehouse architectureTalend T utorialTalend ETL ToolTalend Interview QuestionsFact Table and its TypesInformatica TransformationsInformatica TutorialVIEW ALL Databases What is MySQLMySQL Data TypesSQL JoinsSQL Data TypesWhat is MongoDBMongoDB Interview QuestionsMySQL TutorialSQL Interview QuestionsSQL CommandsMySQL Interview QuestionsVIEW ALL DevOps What is DevOpsDevOps vs AgileDevOps ToolsDevOps TutorialHow To Become A DevOps EngineerDevOps Interview QuestionsWhat Is DockerDocker TutorialDocker Interview QuestionsWhat Is ChefWhat Is KubernetesKubernetes TutorialVIEW ALL Front End Web Development What is JavaScript â€" All You Need To Know About JavaScriptJavaScript TutorialJavaScript Interview QuestionsJavaScript FrameworksAngular TutorialAngular Interview QuestionsWhat is REST API?React TutorialReact vs AngularjQuery TutorialNode TutorialReact Interview QuestionsVIEW ALL Mobile Development Android TutorialAndroid Interview QuestionsAndroid ArchitectureAndroid SQLite DatabaseProgramming Future... Big Data For ETL and Data Warehousing (11 Blogs) Become a Certified Professional AWS Global Infrastructure Introduction to Talend What Is Talend? â€" An Unified Platform For Data IntegrationTalend Architecture â€" Functional Architecture of Talend Open Studio Talend Data Integration Talend ETL Tool â€" Talend Open Studio For Data ProcessingTalend Tutorial â€" Future Of Data Integration Talend Big Data Talend Big Data Tutorial â€" A Revolution In Big Data Talend Interview Questions Top 75 Talend Interview Questions and Answers for 2020Data Warehousing and ETL Topics CoveredData Warehousing (9 Blogs)Big Data For ETL and Data Warehousing (6 Blogs)Informatica PowerCenter 9.X Dev and Admin (8 Blogs)Mastering in Data WareHousing and BI (1 Blogs)SEE MORE Talend Tutorial Future Of Data Integration Last updated on May 26,2020 9.6K Views Swatee Chand Research Analyst at Edureka. A techno freak who likes to explore different... Research Analyst at Edureka. A techno freak who likes to explor e different technologies. Likes to follow the technology trends in market and write about... Bookmark 2 / 2 Blog from Talend Data Integration Become a Certified Professional In todays data-drivenworld a huge amount of data is generated from various organizations, machines, and gadgets, irrespective of their sizes. For example, your mobile, each time you browse the web, some amount of data is generated. Do you know a commercial plane can generate up to500GB of data per hour? I hope now you can imagine how large thisdata is! This is the reason it is known as Big Data. But all of this data is pretty much uselessunless you perform ETL operations on it! Believe me, its certainly not an easy task. Moreover, todays real-time and fast-paced nature of the business, adds to the need of having such a tool which can quickly and easily integrate the systems.Well, this is where Talend comes to the rescue. Through this blog on Talend Tutorial, I will explain how Talend helps to build, test, dep loy, schedule and monitor this data.But before I proceed, let me list down the topics I will be discussing today:What Is Talend?Introduction To Talend Open StudioTOS InstallationTOS GUITalend JobTalend Components and ConnectorsMetadataContext VariablesFirst Job In TalendYou may also go through this recording of Talend Data Integration Tutorial where our Talend Training experts have explained the topics in a detailed manner with examples.Talend Data Integration Tutorial | Talend Online Training | EdurekaThis Edureka video on Talend Data Integration Tutorial will help you in understanding the basic concepts of Talend and getting familiar with the Talend Open Studio which is an open-source software provided by Talend to develop the ETL Jobs.What Is Talend? Talend TutorialTalend isan open source software integration platform/vendor which offers data integration and data management solutions. This company provides various integration software and services for big data, cloud storage, da ta integration, data management, master data management, data quality, data preparation, and enterprise applications. Its headquarters are located in Redwood City, California.Following are the some of the major features of Talend:It is considered to be the next-generation leader in cloud and big data integration software. Itprovides the software that helps companies become data driven by making data more accessible, improving its quality and quickly moving it where its needed for real-time decision making.You can think Talend as a critical infrastructure for this data-drivenworld. Its an open source approach which breaks off the traditional proprietary model by providing the powerful software solutions. It enables the flexibility to meet the needs of all the organizations. Being open source, it is backed by a huge community of the developers. Talend publishes its core modules codes under the GNU Public License or the Apache License. From here, the developers within the community can make changes and enhance the products which in turn will benefit other Talend users.Various products offered by Talend are:Among all the above-shownproducts, Talend Open Studio (TOS) is the main and majorly used. In this Talend tutorial blog, I will be explaining how you can use Talend Open Studio for Data Integration.Introduction To Talend Open Studio (TOS) Talend TutorialTalend Open Studio is an open source project that is based on Eclipse RCP. It supports ETL oriented implementations and is generally provided for the on-premises deployment. It is extensively used for integration between operational systems, ETL processes and data migration. Talend Open Studio for Data Integration is designed in such a way that it can easily combine, convert and update data present at various locations across an organization. This acts as a code generatorwhich produces data transformation scripts and underlying programs in Java. It provides an interactive and user-friendly GUI which lets you acce ss metadata repository containing the definition and configurations for each process performed in Talend. Below is the basic architecture of Talend Open Studio.Lets now try to download and install Talend Open Studio on CentOS. TOS Installation Talend TutorialSTEP 1:Go to: https://www.talend.com/download.STEP 2: Click on Download Free Tool.STEP 3:Again click on Download Free Tool to get the zip file.STEP 4: Now extract the zip file.STEP 5: Now go into the extracted folder and double click onTOS_DI-linux-gtk-x86_64file.STEP 6: Let the installation finish.STEP 7: Click on Create a new project and specify a meaningful name for your project.STEP 8: Click on Finish to go to the Open Studio GUI.STEP 9: Right-click on the Welcome tab and select Close.STEP 10: Now you should be able to see the TOS main page.TOS GUI Talend TutorialNow that you have downloaded and installed Talend Open Studio, let me give you a walkthrough of its GUI. Talend Open Studio consists of four major parts, as shown below.RepositoryThe Repository collects all the technical items which can be used either to describe business models or design Jobs within Talend and displays them in a tree structure. From the Repository, you can access various Business Models, Job Designs, reusable routines, documentation as well as database connections. In other words, the Repositoryacts as a central store for all the elements which are necessary for any Job design or business modelling within a project.Design WindowThis window further consists of the following parts: Workspace: Here you can lay down the designs of your Jobs as well as the business models.Designer Tab: This tab opens bydefault when you create a Job which displays the Job in a graphical mode. Code Tab:This tab helps you in visualizing the code and highlight the possible language errors.PaletteComponent Palette is docked at the top of the design workspace to help you draw the model corresponding to your workflow needs. Depending on your Job or the business model, you can drag and drop various technical components or shapes into your design workspace. There are more than 800 components available for you to choose from.Configuration TabThe configuration tabs are present in the lower half of the design window. There are various configurational tabs available in TOS. Each of these tabs opens a view which displays the properties of the current element in the workspace. Most frequently used configurational tabs are:Job Tab:The Job tab provides various information about the current Job in the designer window including name, version, creation date and time etc.Context TabThe Context tab is used to set context variables and different contexts on which they will be used.Component TabThe Component tab displays all the parameters that are required to configure a component. Basically, it collects all theinformation that is relative to the graphical element selected in the design workspace.Run TabThe Run tab displays the progress of the e xecution of a Job. The logs shown here includes anystart, end and error messages.Here you might ask what is a Job, as I have already used this term quite a few times till now. So, before diving any deeper let me first give you a brief about a Talend Job.Talend Job Talend TutorialA Job in Talend is basically a customer requirement converted into a technical process. Technically, it is a basic executable unit of any process that is built using Talend. As you already know, TOS converts everything into Java codes at the backend. In case of Jobs, each Job is converted into a single Java class. Let me show you how you can create a Job in Talend.Steps:Right-click on the Job Designs in the Repository and select Create job.Specify a meaningful name for your Job along with the purpose and description of it and click on Finish.Once you finish creating a Job, you will get access to the components present in the palette. Now you can drag any component you need from the palette and drop it in th e workspace.But in order to add a component to a Job, first, you need to know what exactly are components, how you can use multiple components together and connect them. So in the next part of this Talend tutorial, I will introduce you to various components and connectors available in Talend.Talend Components And Connectors Talend TutorialLets start with Components.A component is a functional piece which is used to perform a single operation in Talend. On the palette, whatever you can see all are the graphical representation of the components. You can use them with a simple drag and drop. At the backend, a component is a snippet of Java code that is generated as a part of a Job (which is basically Java class). These Java codes are automatically compiledwhen the Job is saved.A Talend Job may include one or more components depending on the requirement. One thing you need to know here is Talend provides more than 800 components from which you can choose from. For the ease of access, a ll these components are generalized to few groups or families. In this Talend tutorial blog, I will introduce you to some of the most important and frequently used components of each family.DatabasesThis family provides Talend components which cover various needs like opening connections, reading and writing tables, committing transactions, performing rollback for error handling etc. More than 40 RDBMS are supported by Talend some of which are MySQL, MS SQL Server, Hive, Amazon, Azure etc. Following are some of the majorly used MySQL components:tMysqlConnection:This component opens a new connection to the database for a current transaction.tMysqlInput:This component reads a database and extracts fields based on the query.tMysqlOutput:This component writes, updates, makes changes or suppresses entries in a database.tMysqlClose:This component closes the transaction committed in the connected database.FileThis family groups together various components which read and write data in all t ypes of files like Delimited, Positional, XML, Excel etc. Moreover, it also provides a number of components which help in performing various tasks like unarchiving, deleting, copying, comparing etc. This family is further divided into subfamilies like Input, Output, and Management. Few majorly used components of this family are:tFileInputDelimited:This component reads a given file row by row with fields separated using some specified character.tFileInputExcel:This component reads an Excel file (.xls or .xlsx) and extracts data line by line.tFileOutputXML:This component outputs the data to a XML type of file.tFileList:This component retrieves a set of files or folders based on a filemask pattern and iterates them.tFileArchive:This component zips one or more files according to the parameters defined and places the archive created in the selected directory.InternetThis family includes all of the components that help in accessing information from the Internet, through various means like Web services, RSS flows, SCP,MOM,Emails, FTP etc. Few of the majorly used components of this family are:tFTPGet:This component helps in retrieving the specified files via an FTP connection.tFTPPut:This component copies the selected files via an FTP connection.tHttpRequest:This component sends an HTTP request to the server end and receives the corresponding response from the server end.tSendMail: This component is used to send emails and attachments to the defined recipients.Logs ErrorsThis family, groups together all the components which are dedicated to catchlog information and handle Job errors. Following are the majorly used components of this family:tLogRow:This component allows you to writerowdata into the Job log file, or to the console window.tLogRowCatcher: This component collects the log data and encapsulates it to pass it on to the defined output.tWarn:This component triggers a warning often caught by the tLogCatcher component for the exhaustive log.tDie:This component s ends a message to atLogCatcherand allows the Job to terminate a Job, with a specifiedExit CodeMiscThis family gathers different miscellaneous components covering various needs like the creation of sets of dummy data rows, buffering data, loading context variables etc. Few important components of this family are:tMsgBox: This component opens a dialogue box with a clickable OK button.tRowGenerator:This component is used to generate as many rows and fields as are required using random values which are taken from a list.OrchestrationThis family includes various components which help to sequence or orchestrate tasks and processing Jobs or SubJobsetc. Majorly used components from this family are:tLoop:This component helps in executing a task or a Job automatically, based on a loop with the specified number of iterations.tPrejob:This component helps in triggering a task required for the execution of a Job.tPostjob:This component helps in triggering a task required after the execution of a Job.tSleep:This component helps in implementing a time off within a Job execution.Now that you know the components, lets quickly take a look at the connectors or the links which help in connecting these components together in a Job.Talend provides various types of connections to enable the communication between the components:RowThe Row connection deals with the actual data flow. Following are the types of Row connections supported by Talend:MainLookupFilterRejectsErrorRejectsOutputUniques/DuplicatesMultiple Input/OutputIterateThe Iterate connection is used to perform a loop on files contained in a directory, on rows contained in a file or on the database entries.Unlike other types of connections, the name of thisIteratelink is read-only.TriggerThe Trigger connection is used to create a dependency between Jobs or SubJobs which are triggered one after the other according to the triggers nature.Trigger connections are generalized in two categories:Subjob TriggersOnSubjobOKOnSubjobErro rRun ifComponent TriggersOnComponentOKOnComponentErrorRun ifLinkTheLinkconnection can be used only with the ELT components. It is used totransfer the table schema information to the ELT mapper component in order to be used in specific DB query statements.Metadata Talend TutorialMetadata in Talend is the definitional data which basically provides information about other data that all are managed within Talend Studio. You can find the Metadata in the Repository area of the TOS. In theRepository Metadata, you can store metadata about the various data sources that you may use. This comes in handy while developing any project as you can use these data sources later in your Jobs, just by dragging an object from the repository and dropping it in the workspace.In the Repository, you can store metadata for various data sources like delimited files, positional file, XML files, database, FTP, Azure, Salesforce etc.Context Variables Talend TutorialContext variables are the user-defined parame ters used by Talend which are passed into a Job at the runtime. These variables may change their values as the Job promotes from Development to Test and Production environment. So, once these variables are setcorrectly for each environment, you can execute a Job easily in any of these environments.Another use of context variables is to define the values which are commonly used within a project. You can create the context variables in three ways:Embedded Context VariablesThese context variables are embedded in the Job and are configured much like any other component parameters in the Context Tab below the Job Designer.Repository Context VariablesThese are created when context variables are used or needed in more than one Job. They are centrally maintained in the repository allowing them generally accessible.External Context VariablesExternal context variables are those context variables which are held in an external file and loaded into the Studio job at the run-time.Now, I think you are ready to design your First job in Talend.In the next section of this Talend tutorial blog, I will show you a step by step demonstration of a simple Talend Job which you can easily execute.First Job In Talend Talend TutorialFollowing is a demo in which first you will be establishing a connection with the database, read data from two different external excel files, merge them and then insert it into the database table. Then in a new excel file write the new table contents. Finally, close the connection once the transfer is complete.Lets see how to execute it, step by step:STEP 1:In this demo, I am using external context file for database details. In order to do so, first, you need to create a context file with all the necessary database details.STEP 2: Create a new Job. Got to its Contexts tab and add the following details:STEP 3:Now, add a PreJob and a tMysqlConnection components in the workspace and link them together as shown below. This will establish the connection with the database before the actual Job is executed. Then go to the Component tab of tMysqlConnection component and add the necessary details:STEP 4:Add two tFileInputExcel files and a tMap component in the workspace and link them as shown.STEP 5:Now go to the Repository and expand Metadata section. Right click on File Excel and select Create File Excel and then provide the necessary details as shown below. Once done click on Next.STEP 6:Provide the source file path and click on Next.STEP 7:Check on Header to skip the header row (if applicable). Click on Next.STEP 8:Finally provide a name for the schema and click on Finish.STEP 9:Go to the Component tab of the tFileInputExcel component. Select the Property Type as Repository and select the metadata, you just created.STEP 10:Repeat the same for the other input file.STEP 11:Double-clickon the tMap component and map the input and output tables as shown:STEP 12:Add tMysqlOutput and tFileOutputExcel components and link them as shown:STEP 13:Go t o the component tab of tMysqlOutput and enter the details as shown:STEP 14:Go to the component tab of tFileOutputExcel and provide the details as shown:STEP 15:Finally to finish the job, add a Postjob and a tMysqlClose component as shown.STEP 16:Go to the Component tab of the tMysqlClose component and select the connection you need to close.STEP 17:Now go to the Run tab and execute the job.So, this brings us to the end of the blog on Talend Tutorial. I tried my best to keep the concepts short and clear. Hope it helped you in understanding Talend and its various features. Regarding the demo, if you need the datasets for the practice, all you need to do is drop a comment.If you found this Talend tutorialblog, relevant,check out the Talend for DI and Big Data Certification Trainingby Edureka,a trusted online learning companywith a network of more than250,000satisfied learnersspread acrossthe globe. The Edureka Talend for DI and Big Data Certification Training course helps you to master Talend and Big Data Integration Platform and easily integrate all your data with your Data Warehouse and Applications, or synchronize data between systems.Got a question for us? Please mention it in the comments section and we will get back to you.Recommended videos for you Introduction To Data Warehousing Watch Now Informatica PowerCenter : Agile Data Integration Tool Watch Now Talend: The Non-Programmers Swiss Knife for Big Data-I Watch Now Talend: The Non-Programmers Swiss Knife for Big Data-II Watch Now Management in Informatica PowerCenter Watch Now Simplifying Big Data Using Talend Watch Now Management in Informatica Powercenter Watch Now ETL Using Informatica Power Center Watch Now Talend for Big Data:Secret Key to Hadoop Watch Now Designing and Monitoring in Informatica Powercenter Watch Now ETL using Big Data Talend Watch Now Informatica Capabilities As An ETL Tool Watch NowRecommended blogs for you Data Warehousing Interview Questions And Answers You Must Prepare In 2020 Read Article Shedding some Light on Apache Solr Read Article Informatica Certification: All there is to know Read Article Informatica Transformations: The Heart and Soul of Informatica PowerCenter Read Article Talend ETL Tool Talend Open Studio For Data Processing Read Article Informatica Interview Questions Part 2 For 2020: Scenario-Based Interview Questions Read Article A Brief on Data Warehouse Read Article Talend Tutorial Future Of Data Integration Read Article Career Progression With Informatica: All You Need To Know Read Article Informatica ETL: A Beginner’s Guide To Understanding ETL Using Informatica PowerCenter Read Article Top Informatica Interview Questions You Must Prepare In 2020 Read Article Types Of Dimension Table Read Article Data Warehousing And Business Intelligence Career Path: Bag Data Warehousing And Data Mining Jobs Read Article Talend Big Data Tutorial A Revolution In Big Data Read Article Dimension Table in Data Warehousing Read Article Architecture of a Data Warehouse Read Article Top 75 Talend Interview Questions and Answers for 2020 Read Article I Got Data In My Jewellery Box Read Article What Is Talend? An Unified Platform For Data Integration Read Article Fact Table and its Types in Data Warehousing Read Article Comments 0 Comments Trending Courses in Data Warehousing and ETL Informatica Training Certification16k Enrolled LearnersWeekendLive Class Reviews 5 (6200)