fg-data-profiling
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
What is fg-data-profiling?
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Overview of fg-data-profiling
What it is
fg-data-profiling is an open-source Python library that provides one-line exploratory data analysis (EDA) and data quality profiling for Pandas and Spark DataFrames. It extends the functionality of pandas df.describe() by generating a comprehensive analysis report that can be exported to HTML, JSON, or displayed as a widget in Jupyter Notebooks. The package was formerly known as ydata-profiling and has been renamed to fg-data-profiling; the old package will no longer receive updates or bug fixes.
Key features
- Type inference: automatic detection of columns' data types (Categorical, Numerical, Date, etc.)
- Warnings: summary of problems/challenges in the data such as missing data, inaccuracies, and skewness
- Univariate analysis: descriptive statistics (mean, median, mode) and distribution histograms
- Multivariate analysis: correlations, missing data analysis, duplicate rows, and pairwise interaction visualizations
- Time-Series analysis: auto-correlation, seasonality, ACF and PACF plots
- Text analysis: most common categories (uppercase, lowercase, separator), scripts (Latin, Cyrillic), and blocks (ASCII, Cyrillic)
- File and Image analysis: file sizes, creation dates, dimensions, indication of truncated images, and EXIF metadata
- Compare datasets: one-line solution for comparing two datasets
- Flexible output formats: HTML report, JSON for automated systems, and Jupyter Notebook widget
Use cases
fg-data-profiling can be used for comparing multiple versions of the same dataset, profiling a time-series dataset with a single line of code, profiling large datasets (with configuration tips), handling sensitive data by generating mindful reports, complementing reports with dataset metadata and data dictionaries, and customizing the report's appearance. It also supports use inside Jupyter Notebooks via widgets or embedded HTML, and can be used from the command line for standard CSV files.
Getting started
Install the package:pip install fg-data-profilingOr via conda:conda install -c conda-forge fg-data-profilingTo generate a profiling report:pythonimport numpy as npimport pandas as pdfrom data_profiling import ProfileReportdf = pd.DataFrame(np.random.rand(100, 5), columns=[“a”, “b”, “c”, “d”, “e”])profile = ProfileReport(df, title=“Profiling Report”)
License and pricing
The license is MIT. The package is open-source and free to use.
這是什麼
fg-data-profiling 是一個開源的 Python 函式庫,提供一行程式碼即可對 Pandas 和 Spark DataFrame 進行探索式資料分析(EDA)與資料品質剖析。它擴展了 pandas df.describe() 的功能,能產生全面的分析報告,並可匯出為 HTML、JSON 格式,或在 Jupyter Notebook 中以 widget 顯示。此套件原名為 ydata-profiling,現已更名為 fg-data-profiling;舊套件將不再接收更新或錯誤修復。
主要功能
- 型別推斷:自動偵測欄位的資料型別(類別、數值、日期等)
- 警告:摘要資料中的問題與挑戰,例如缺失值、不準確性與偏態
- 單變量分析:描述性統計(平均數、中位數、眾數)與分佈直方圖
- 多變量分析:相關性、缺失值分析、重複列與變數配對互動視覺化
- 時間序列分析:自相關、季節性、ACF 與 PACF 圖
- 文字分析:最常見類別(大寫、小寫、分隔符)、文字系統(拉丁、西里爾)與區塊(ASCII、西里爾)
- 檔案與圖片分析:檔案大小、建立日期、尺寸、截斷圖片指示與 EXIF 中繼資料
- 資料集比較:一行程式碼即可比較兩個資料集
- 彈性輸出格式:HTML 報告、JSON(適用於自動化系統)與 Jupyter Notebook widget
使用情境
fg-data-profiling 可用於比較同一資料集的多個版本、以一行程式碼剖析時間序列資料集、剖析大型資料集(需搭配設定建議)、處理敏感資料(產生注意隱私的報告)、補充資料集中繼資料與資料字典,以及自訂報告外觀。它也支援在 Jupyter Notebook 中以 widget 或嵌入 HTML 使用,並可透過命令列處理標準 CSV 檔案。
快速開始
安裝套件:pip install fg-data-profiling或透過 conda:conda install -c conda-forge fg-data-profiling產生剖析報告:pythonimport numpy as npimport pandas as pdfrom data_profiling import ProfileReportdf = pd.DataFrame(np.random.rand(100, 5), columns=[“a”, “b”, “c”, “d”, “e”])profile = ProfileReport(df, title=“Profiling Report”)
授權與價格
授權為 MIT。此套件為開源且免費使用。
Based on the project's public README and repository metadata.