|
Tidal™ Enterprise Adapter for IBM® WebSphere® DataStage
Download the PDF version of this document
IBM WebSphere DataStage is a powerful ETL solution that integrates data from
diverse sources. It enables the collection, integration, and transformation of high volumes of
simple and highly complex data that can be received real-time or on a scheduled basis. The Tidal
Enterprise Scheduler Adapter for DataStage Enterprise Edition supports the DataStage ETL processes
by enabling customers to consolidate ETL scheduling automation into a single point of control
for viewing and managing the complex job schedules involved in ETL operations. Even more important,
it allows enterprises to manage ETL scheduling within the context of the larger enterprise operations
schedule. This increases overall operational efficiency, and reduces the risk of errors because all
dependencies are easily accommodated in the enterprise schedule.
DataStage can target a wide array of heterogeneous data sources in a single
job, including text files; complex data structures in XML; ERP systems such as SAP and PeopleSoft;
most databases; web services; and business intelligence tools. However, it is not optimized for
consolidating, streamlining, and controlling all enterprise-wide job scheduling. Tidal’s enterprise
scheduling adapter for DataStage has been developed to address the specific scheduling requirements
of DataStage jobs. It allows IT to schedule DataStage jobs seamlessly within consolidated schedules
that operate across enterprise systems and applications.
The most important aspect of the adapter is that it provides DataStage jobs with
the extensive set of rich scheduling features that are available to all Tidal Enterprise Scheduler
jobs. These include calendar and/or event-based scheduling, alerting and centralized operational
management via the Scheduler’s job activity console. Also, features specific to DataStage scheduling
have been developed to provide additional value for managing the DataStage application.
Simplification through Consolidation
Tidal’s DataStage adapter allows organizations to add complex scheduling functions
to their ETL processes and to consolidate multiple schedulers into one centralized console. The
enterprise adapter, in conjunction with Tidal Enterprise Scheduler, can be used to define, launch,
and monitor DataStage requests within its own environment.
Once Tidal’s DataStage adapter is installed, minimal effort is required to define
a DataStage job within Tidal Enterprise Scheduler. There are only three simple steps involved in
scheduling jobs:
- Picking the path to the Tidal DataStage bridge program from a list of Tidal
Scheduler variables
- Entering the DataStage job name
- Selecting an environment file containing global configuration data from a list
of Tidal Scheduler variables
By taking advantage of the enterprise adapter, organizations can create a job
definition to manage the most simple or the most complex request sets defined within the DataStage
data integration environment. Tidal Enterprise Scheduler allows users to develop process streams
and manage the information flow from a centralized console, thereby giving them greater control
over all aspects of a job stream.
Tidal also provides a more convenient, consolidated way to examine the sequence
of events that otherwise would have to be pieced together if using the DataStage console alone. Tidal
accomplishes this through output that includes a collection of logs from all jobs that are run from
inside a sequencer job. The merged log files can be observed in real time and/or upon completion.
Job Definition – The adapter allows DataStage jobs to be defined and
command parameters set easily within the Tidal scheduler. efficiency and data reliability
Tidal Enterprise Scheduler provides virtually unlimited flexibility in developing
process streams. Users can develop complex job streams that utilize the adapter to merge any system
process and ERP application process into a cohesive, centrally managed job stream. This provides error
detection, correction, and management across the enterprise. Since the DataStage request set is defined
just like any other Tidal Enterprise Scheduler job, users can take full advantage of job dependencies,
file dependencies, variable dependencies, calendars, events, and email notifications.
How It Works
The DataStage adapter uses a bridge program and required DataStage libraries.
The DataStage server and repository can be on any machine in the data center that DataStage supports.
The creation of a DataStage job from within Tidal Enterprise Scheduler is done in the same way as
creating any other job. A Tidal job definition defines the internal request set information defined
within DataStage and is used by the Enterprise Adapter to launch the DataStage request.
Job Details – A consolidated view of details for a sequence of jobs
makes it easy to track progress and identify errors quickly.
Once the request is made to DataStage, the Tidal adapter monitors the process
until it completes. The adapter also captures the DataStage process exit code, returns it to Tidal
Enterprise Scheduler and provides additional log information about the request. In addition, the
adapter allows users to maintain complete control over each request that is launched. The request
can be cancelled, aborted, or rerun from Tidal Enterprise Scheduler, a feature that provides extensive
user control over the environment.
The adapter launches processes based on defined calendars and schedules, and
monitors and controls those processes as they execute. In addition, it provides error control and
failure notification to enhance the overall business processes that drive an organization’s data
processing requirements.
The adapter supports dynamic parameter updates using a parameter file or command-line
overrides with values that can come from standard Tidal variables, custom supplied and/or calculated
values or values generated and externally set by other applications or derived from the output of
predecessor jobs.
In summary, the IBM Websphere DataStage adapter leverages Tidal Enterprise
Scheduler’s cutting-edge job scheduling technology to automate data feeds into business intelligent
products.
|