Data Analysis with Python and pandas

01Getting started with pandas and loading data

pandas has two core structures: Series (one column) and DataFrame (a table). Install it with pip install pandas; the official docs live at pandas.pydata.org/docs. We use a simple sales dataset here; if you have no file, we build the same data in code so every example runs.

import pandas as pd

# In real work: df = pd.read_csv("sales.csv", parse_dates=["date"])
df = pd.DataFrame({
    "date": ["2026-01-03", "2026-01-15", "2026-02-02", "2026-02-20", "2026-03-05", "2026-03-18"],
    "city": ["Tabriz", "Tehran", "Tabriz", None, "Tehran", "Tabriz"],
    "product": ["rug", "rug", "kilim", "rug", "kilim", "rug"],
    "qty": [2, 1, 3, 1, None, 4],
    "price": [1200, 1500, 800, 1500, 800, 1200],
})
df["date"] = pd.to_datetime(df["date"])

print(df.shape)
print(df.dtypes)
print(df.head())

For files with Persian text, check the encoding: pd.read_csv("file.csv", encoding="utf-8-sig") usually fixes Excel exports. Read big files lighter with usecols and dtype.

02Knowing the data before analysing it

Before any calculation, run a few fixed commands. It takes minutes and prevents wrong conclusions from bad data:

• df.info(): column types and non-null counts.
• df.describe(): mean, min and max; reveals implausible values such as negative prices.
• df.isna().sum(): missing values per column.
• df["city"].value_counts(dropna=False): duplicates and typos in categorical columns.
• df.duplicated().sum(): duplicate rows.

03Cleaning: missing values, types and duplicates

Cleaning is a set of decisions, not just commands. Dropping a row, filling with the mean or keeping "unknown" each leads to a different result; decide based on what the column means, and write the decision down.

clean = (
    df
    .drop_duplicates()
    .assign(
        city=lambda d: d["city"].fillna("unknown").str.strip().str.title(),
        qty=lambda d: d["qty"].fillna(0).astype(int),
        product=lambda d: d["product"].astype("category"),
    )
    .assign(revenue=lambda d: d["qty"] * d["price"])
)
print(clean)

The chain above (method chaining) keeps the steps in order and readable, and leaves the original DataFrame untouched. Note that filling a missing quantity with 0 here means "no sale recorded"; if it means "unknown", drop the row instead.

04Grouping and summarising with groupby

Most business questions have the shape "for each X, how much is Y", and the answer is groupby:

by_city = (
    clean.groupby("city", as_index=False)
         .agg(orders=("qty", "size"), units=("qty", "sum"), revenue=("revenue", "sum"))
         .sort_values("revenue", ascending=False)
)
print(by_city)

pivot = clean.pivot_table(index="city", columns="product", values="revenue",
                          aggfunc="sum", fill_value=0, observed=True)
print(pivot)

Named aggregation (name=("column", "func")) keeps the output readable with no renaming. pivot_table reshapes the same result into a matrix that suits city-versus-product comparison. If you present results in a web report, our data analysis service page describes how we work.

05Combining tables with merge

Real data usually spans several tables: orders, customers, products. merge is the equivalent of a SQL JOIN, and you choose the join type with how.

cities = pd.DataFrame({
    "city": ["Tabriz", "Tehran", "Shiraz"],
    "region": ["North-West", "Centre", "South"],
})

# validate catches accidental many-to-many joins that silently multiply rows
merged = clean.merge(cities, on="city", how="left", validate="many_to_one")
print(merged[["city", "region", "revenue"]])
print(merged["region"].isna().sum(), "rows without a region")

If your data lives in a database, it is often better to do the heavy join there in SQL and bring only the result into pandas; query design also comes up in scalable API design.

06Time series: dates, resampling and trends

Once the date column is a real datetime, grouping by month or week is a one-liner. Convert the type at the very start, not in the middle of the analysis.

monthly = (
    clean.set_index("date")["revenue"]
         .resample("MS")          # month start
         .sum()
         .to_frame("revenue")
)
monthly["change_pct"] = monthly["revenue"].pct_change() * 100
monthly["rolling_2m"] = monthly["revenue"].rolling(2).mean()
print(monthly)

pct_change returns an empty value for the first month, which is expected. With little data keep the rolling window short and avoid over-interpreting; a few months are not enough to claim a "trend".

07Common pandas pitfalls

• Looping over rows: iterrows() is slow. Compute columns vectorised: df["a"] * df["b"].
• SettingWithCopyWarning: do not assign into a slice; use .loc or .copy().
• Column types: numbers read as strings will not sum; pd.to_numeric(col, errors="coerce") turns bad values into NaN so you can see where they are.
• Memory: convert repetitive text columns to category and read very large files in pieces with chunksize.

Analysis usually ends in a chart or an output file: by_city.to_csv("report.csv", index=False) or df.plot(kind="bar") (via Matplotlib). When publishing to a site, keep page load speed in mind.

08A reproducible-analysis checklist and next steps

1. Keep raw data untouched and do cleaning in code.
2. Set column types at load time.
3. Check row counts after every merge.
4. Write assumptions (such as "missing means 0") as code comments.
5. Put every step in one script or notebook so it reruns on new data.

With this base in place, the natural next step is models; an introduction to machine learning starts from exactly these clean tables. For the language fundamentals, the Python documentation is a solid reference. And if your business data is ready to analyse, talk to us.