User Guide - Revision history

172.18.0.10: typo on UNITA (name), that was written UINTA

2025-07-14T15:35:23Z

typo on UNITA (name), that was written UINTA

← Older revision		Revision as of 15:35, 14 July 2025
Line 7:		Line 7:

	===Data Collection Methodology===		===Data Collection Methodology===
	Two ways for collecting the data were implemented for the '''Impact Observatory'''. In the automatic path, users simply upload CSV files to our [https://unitapedia.univ-unita.eu/minio/buckets MinIO] object-storage service; from there, a suite of '''Apache HOP''' jobs automatically ingests, transforms, and loads the data into our Datalake, ensuring updated metrics with zero manual intervention. In the manual path, ~~UINTA~~ offices log into our [https://unitapedia.univ-unita.eu/strapi/ Strapi] CMS and complete pre‐built indicator forms (e.g., entering dates, numeric values, and descriptions). Once submitted, each entry is stored in our '''PostgreSQL''' database, tracked with metadata (timestamps, author, version), and immediately available for reporting. This dual approach guarantees both high volume automation and flexible, data entry, so all the indicators stay accurate and updated.		Two ways for collecting the data were implemented for the '''Impact Observatory'''. In the automatic path, users simply upload CSV files to our [https://unitapedia.univ-unita.eu/minio/buckets MinIO] object-storage service; from there, a suite of '''Apache HOP''' jobs automatically ingests, transforms, and loads the data into our Datalake, ensuring updated metrics with zero manual intervention. In the manual path, UNITA offices log into our [https://unitapedia.univ-unita.eu/strapi/ Strapi] CMS and complete pre‐built indicator forms (e.g., entering dates, numeric values, and descriptions). Once submitted, each entry is stored in our '''PostgreSQL''' database, tracked with metadata (timestamps, author, version), and immediately available for reporting. This dual approach guarantees both high volume automation and flexible, data entry, so all the indicators stay accurate and updated.

	[[File:Data_Collection.png\|frameless\|996px\|center\|Data Injection]]		[[File:Data_Collection.png\|frameless\|996px\|center\|Data Injection]]

Antonio De Jesus Menchaca Martinez: /* Strapi Manual Data Injection Steps */

2025-06-13T09:30:16Z

Strapi Manual Data Injection Steps

← Older revision		Revision as of 09:30, 13 June 2025
Line 18:		Line 18:
	* Select the indicator entry form to fulfill.		* Select the indicator entry form to fulfill.
	* Under the “COLLECTION TYPES” panel the respective forms will be displayed according to the user roles. We defined three user roles with different permissions:		* Under the “COLLECTION TYPES” panel the respective forms will be displayed according to the user roles. We defined three user roles with different permissions:
	** a) UNITAOffice		:: a) UNITAOffice
	** b) TaskLeader		:: b) TaskLeader
	** c) ProjectManager		:: c) ProjectManager
	* Once you select the form entry to fulfill. Click on “Create new entry”.		* Once you select the form entry to fulfill. Click on “Create new entry”.
	* Enter the required data manually.		* Enter the required data manually.

Antonio De Jesus Menchaca Martinez: /* Data Collection Methodology */

2025-06-13T09:29:45Z

Data Collection Methodology

← Older revision		Revision as of 09:29, 13 June 2025
Line 7:		Line 7:

	===Data Collection Methodology===		===Data Collection Methodology===
	Two ways for collecting the data were implemented for the '''Impact Observatory'''. In the automatic path, users simply upload CSV files to our [https://unitapedia.univ-unita.eu/minio/buckets MinIO] object-storage service; from there, a suite of '''~~Pentaho~~''' jobs automatically ingests, transforms, and loads the data into our Datalake, ensuring updated metrics with zero manual intervention. In the manual path, UINTA offices log into our [https://unitapedia.univ-unita.eu/strapi/ Strapi] CMS and complete pre‐built indicator forms (e.g., entering dates, numeric values, and descriptions). Once submitted, each entry is stored in our '''PostgreSQL''' database, tracked with metadata (timestamps, author, version), and immediately available for reporting. This dual approach guarantees both high volume automation and flexible, data entry, so all the indicators stay accurate and updated.		Two ways for collecting the data were implemented for the '''Impact Observatory'''. In the automatic path, users simply upload CSV files to our [https://unitapedia.univ-unita.eu/minio/buckets MinIO] object-storage service; from there, a suite of '''Apache HOP''' jobs automatically ingests, transforms, and loads the data into our Datalake, ensuring updated metrics with zero manual intervention. In the manual path, UINTA offices log into our [https://unitapedia.univ-unita.eu/strapi/ Strapi] CMS and complete pre‐built indicator forms (e.g., entering dates, numeric values, and descriptions). Once submitted, each entry is stored in our '''PostgreSQL''' database, tracked with metadata (timestamps, author, version), and immediately available for reporting. This dual approach guarantees both high volume automation and flexible, data entry, so all the indicators stay accurate and updated.

	[[File:Data_Collection.png\|frameless\|996px\|center\|Data Injection]]		[[File:Data_Collection.png\|frameless\|996px\|center\|Data Injection]]

Antonio De Jesus Menchaca Martinez: /* ETL Pipeline Tutorials */

2025-06-11T11:09:06Z

ETL Pipeline Tutorials

← Older revision		Revision as of 11:09, 11 June 2025
Line 57:		Line 57:
	= ETL Pipeline Tutorials =		= ETL Pipeline Tutorials =

	This three-part tutorial series walks you through the complete ETL (Extract, Transform, Load) process used to feed structured data into the UNITApedia platform. Using [https://unitapedia.univ-unita.eu/strapi Strapi], [https://unitapedia.univ-unita.eu/hop Apache HOP], and [https://unitapedia.univ-unita.eu/pga PostgreSQL], we show how to create and manage data flows from form input to dynamic visualization.		This three-part tutorial series walks you through the complete '''ETL (Extract, Transform, Load)''' process used to feed structured data into the UNITApedia platform. Using [https://unitapedia.univ-unita.eu/strapi Strapi], [https://unitapedia.univ-unita.eu/hop Apache HOP], and [https://unitapedia.univ-unita.eu/pga PostgreSQL], we show how to create and manage data flows from form input to dynamic visualization.

	=== Part I – Strapi: Data Collection Form ===		=== Part I – Strapi: Data Collection Form ===

Antonio De Jesus Menchaca Martinez: /* ETL Piepeline Tutorials */

2025-06-11T11:08:53Z

ETL Piepeline Tutorials

← Older revision		Revision as of 11:08, 11 June 2025
Line 55:		Line 55:
	This tutorial is part of our documentation for the '''Impact Observatory''' data infrastructure and is useful for both technical and non-technical users setting up ETL pipelines or data access jobs.		This tutorial is part of our documentation for the '''Impact Observatory''' data infrastructure and is useful for both technical and non-technical users setting up ETL pipelines or data access jobs.

	=ETL ~~Piepeline~~ Tutorials=		= ETL Pipeline Tutorials =
	===Part I - Strapi===
	~~As a~~ first part ~~of our ETL pipeline process tutorial~~ we ~~will teach you~~ how to create a form and entries ~~using Strapi~~.		This three-part tutorial series walks you through the complete ETL (Extract, Transform, Load) process used to feed structured data into the UNITApedia platform. Using [https://unitapedia.univ-unita.eu/strapi Strapi], [https://unitapedia.univ-unita.eu/hop Apache HOP], and [https://unitapedia.univ-unita.eu/pga PostgreSQL], we show how to create and manage data flows from form input to dynamic visualization.

			=== Part I – Strapi: Data Collection Form ===
			In this first part, we show how to use [https://unitapedia.univ-unita.eu/strapi Strapi] to create a structured form for data collection. You'll learn how to configure content types, add validation rules, and publish entries that will later be extracted and processed in our pipeline.

	<html>		<html>
	<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="~~Synthesia video player~~ - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>		<div style="position: relative; overflow: hidden; padding-top: 56.25%;">
			<iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Strapi Tutorial - ETL Pipeline Part I" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe>
			</div>
	</html>		</html>

	===Part II - Apache HOP===		=== Part II – Apache HOP: Transformation Pipeline ===
	~~In the second~~ part ~~of our ETL~~ pipeline process ~~tutorial we will teach you how to create~~ a ~~simple ETL~~ transformation ~~using Apache HOP~~.		This part focuses on [https://unitapedia.univ-unita.eu/hop Apache HOP], where we create a basic transformation pipeline to extract data from [https://unitapedia.univ-unita.eu/strapi Strapi], process and filter it, and then store it in a [https://unitapedia.univ-unita.eu/pga PostgreSQL] datamart. This transformation prepares the data for use in reporting and analysis tools.

	<html>		<html>
	<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/7b430445-c85f-4c28-b1f0-a2c6318d305b" loading="lazy" title="~~Synthesia video player~~ - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>		<div style="position: relative; overflow: hidden; padding-top: 56.25%;">
			<iframe src="https://share.synthesia.io/embeds/videos/7b430445-c85f-4c28-b1f0-a2c6318d305b" loading="lazy" title="Apache HOP Tutorial - ETL Pipeline Part II" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe>
			</div>
	</html>		</html>

	===Part III ~~- Mediawiki~~===		=== Part III – MediaWiki: Data Integration ===
	In the ~~third~~ part ~~of our ETL pipeline process tutorial~~ we ~~will teach you how~~ to ~~import~~ the data on our ~~mediawiki~~ site ~~using External Data extension~~.		In the final part, we use the External Data extension in MediaWiki to fetch and display the transformed data on our UNITApedia site. You'll learn how to configure database queries, create pages dynamically, and format the output for semantic use. The data displayed here originates from the [https://unitapedia.univ-unita.eu/pga PostgreSQL] datamart populated by [https://unitapedia.univ-unita.eu/hop Apache HOP] from [https://unitapedia.univ-unita.eu/strapi Strapi] entries.

	<html>		<html>
	<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/99bed61a-fabc-4682-88ff-a29e687abfe4" loading="lazy" title="~~Synthesia video player~~ - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>		<div style="position: relative; overflow: hidden; padding-top: 56.25%;">
			<iframe src="https://share.synthesia.io/embeds/videos/99bed61a-fabc-4682-88ff-a29e687abfe4" loading="lazy" title="MediaWiki Integration - ETL Pipeline Part III" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe>
			</div>
	</html>		</html>

Antonio De Jesus Menchaca Martinez: /* ETL Piepeline Tutorials */

2025-06-11T09:46:19Z

ETL Piepeline Tutorials

← Older revision		Revision as of 09:46, 11 June 2025
Line 66:		Line 66:
	<html>		<html>
	<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/7b430445-c85f-4c28-b1f0-a2c6318d305b" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>		<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/7b430445-c85f-4c28-b1f0-a2c6318d305b" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>
			</html>

			===Part III - Mediawiki===
			In the third part of our ETL pipeline process tutorial we will teach you how to import the data on our mediawiki site using External Data extension.
			<html>
			<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/99bed61a-fabc-4682-88ff-a29e687abfe4" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>
	</html>		</html>

Antonio De Jesus Menchaca Martinez: /* Part I - Strapi */

2025-06-11T09:21:33Z

Part I - Strapi

← Older revision		Revision as of 09:21, 11 June 2025
Line 57:		Line 57:
	=ETL Piepeline Tutorials=		=ETL Piepeline Tutorials=
	===Part I - Strapi===		===Part I - Strapi===
	As a first part of our ETL pipeline process we will teach you ~~in this first part of the tutorial,~~ how to create a form and entries using Strapi.		As a first part of our ETL pipeline process tutorial we will teach you how to create a form and entries using Strapi.
	<html>		<html>
	<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>		<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>
			</html>

			===Part II - Apache HOP===
			In the second part of our ETL pipeline process tutorial we will teach you how to create a simple ETL transformation using Apache HOP.
			<html>
			<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/7b430445-c85f-4c28-b1f0-a2c6318d305b" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>
	</html>		</html>

Antonio De Jesus Menchaca Martinez: /* Part I */

2025-06-11T09:15:45Z

Part I

← Older revision		Revision as of 09:15, 11 June 2025
Line 56:		Line 56:

	=ETL Piepeline Tutorials=		=ETL Piepeline Tutorials=
	===Part I===		===Part I - Strapi===
	As a first part of our ETL pipeline process we will teach you in this first part of the tutorial, how to create a form and entries using Strapi.		As a first part of our ETL pipeline process we will teach you in this first part of the tutorial, how to create a form and entries using Strapi.
	<html>		<html>
	<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>		<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>
	</html>		</html>

Antonio De Jesus Menchaca Martinez: /* Part I */

2025-06-11T09:15:23Z

Part I

← Older revision		Revision as of 09:15, 11 June 2025
Line 56:		Line 56:

	=ETL Piepeline Tutorials=		=ETL Piepeline Tutorials=
	=Part I=		===Part I===
	As a first part of our ETL pipeline process we will teach you in this first part of the tutorial, how to create a form and entries using Strapi.		As a first part of our ETL pipeline process we will teach you in this first part of the tutorial, how to create a form and entries using Strapi.
	<html>		<html>
	<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>		<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>
	</html>		</html>

Antonio De Jesus Menchaca Martinez: /* ETL Piepeline Tutorials */

2025-06-11T09:15:04Z

ETL Piepeline Tutorials

← Older revision		Revision as of 09:15, 11 June 2025
Line 56:		Line 56:

	=ETL Piepeline Tutorials=		=ETL Piepeline Tutorials=
			=Part I=
			As a first part of our ETL pipeline process we will teach you in this first part of the tutorial, how to create a form and entries using Strapi.
	<html>		<html>
	<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>		<div style="position: relative; overflow: hidden; padding-top: 56.25%;"><iframe src="https://share.synthesia.io/embeds/videos/b32a5947-b6e1-4ba5-b030-5cb5eaa315fc" loading="lazy" title="Synthesia video player - v2" allowfullscreen allow="encrypted-media; fullscreen;" style="position: absolute; width: 100%; height: 100%; top: 0; left: 0; border: none; padding: 0; margin: 0; overflow:hidden;"></iframe></div>
	</html>		</html>