Background

It went like this.

One night I was browsing the Bilibili Game Center and noticed that some games had high ratings but low download counts. Others had average ratings but high download counts.

I wondered whether there were any patterns among these data points.

So I wrote this scraper.

Project Overview

This project does something simple: it scrapes data from the Bilibili Game Center, then performs analysis and visualization.

The scraped data includes game names, genres, platforms, ratings, follower counts, download counts, ranking data, and more.

The analysis covers genre distribution, platform distribution, rating distribution, the relationship between follower counts and downloads, revenue estimation, and more.

Data Acquisition Modes

The project supports three data acquisition modes:

Mode Description Use Case
demo Built-in simulated data for 88 real domestic games Development and debugging, offline demos
web Parse from Bilibili Game Center HTML pages Actual data collection
api Call Bilibili API endpoints Actual data collection

demo mode is for people who don’t want to scrape data and just want to see what the analysis looks like. It includes data for 88 real domestic games. Although it isn’t the latest, it’s enough for a demo.

web mode gets data by parsing HTML pages. It doesn’t require an API key, but it’s a bit slower.

api mode calls Bilibili’s API directly. It’s fast, but you may need to deal with anti-scraping measures.

Revenue Estimation

The most interesting part of this project is revenue estimation.

The Bilibili Game Center does not publish game revenue data, so I can only estimate it through other metrics. The metrics I used are follower counts and ratings.

Follower counts represent a game’s popularity, and ratings represent its quality. A game with high popularity and high quality is very likely to have decent revenue.

The estimation formula is simple: revenue = follower count * rating coefficient * platform coefficient. This formula is not precise, but it can show the general trend.

Visualization

The project generates 7 types of charts:

  1. Game genre distribution pie chart
  2. Platform distribution bar chart
  3. Rating distribution histogram
  4. Follower count vs. download count scatter plot
  5. Estimated revenue ranking chart
  6. Popularity ranking TOP10
  7. Top-grossing ranking TOP10

These charts are generated with Matplotlib. The styling is average, but the data is accurate. If you want better-looking charts, you can switch to ECharts or Plotly yourself.

Some Findings

After scraping the data, I found some interesting things:

  1. Anime-style games generally have higher ratings. This may be because anime-style game players are more willing to give high scores.

  2. For indie games, follower counts and download counts are not proportional. Some indie games have high follower counts but low download counts. This may be because players only bookmark them and don’t actually play them.

  3. Popular games in the Bilibili Game Center are not quite the same as popular games on the App Store. Bilibili users prefer anime-style, indie, and niche games.

  4. Games on the pre-registration chart perform unevenly after launch. A high pre-registration count doesn’t mean the game is good; it may just mean the marketing was good.

These findings aren’t particularly deep insights, but I find them quite interesting.

Project Structure

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
bgame/
├── main.py                 # 主入口
├── bili_game_scraper/      # 爬虫模块
│   ├── spider.py           # 爬虫引擎
│   ├── demo_data.py        # 模拟数据生成器
│   └── parsers/            # 解析器
├── analysis/               # 分析模块
│   ├── statistics.py       # 统计分析
│   └── visualization.py    # 可视化
├── storage/                # 存储模块
│   └── sqlite_storage.py   # SQLite 存储
├── scheduler/              # 调度模块
│   └── scheduler.py        # 定时调度
├── config/                 # 配置模块
│   └── settings.py         # 配置管理
├── data/                   # 数据目录
├── charts/                 # 图表目录
└── exports/                # 导出目录

Quick Start

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 安装依赖
pip install -r requirements.txt

# 一键执行完整流程：爬取→分析→可视化→导出
python main.py all

# 只爬取数据
python main.py crawl --max-pages 5

# 只生成分析报告
python main.py analyze --save

# 只生成可视化图表
python main.py visualize

# 启动定时爬取
python main.py schedule --interval 6

Pitfalls I Encountered

Pitfall 1: Anti-Scraping

Bilibili has anti-scraping mechanisms, and making requests too frequently can get your IP blocked. I added random delays and User-Agent rotation, but sometimes I still got blocked.

Pitfall 2: Data Format

The data format in the Bilibili Game Center is not very consistent. Some games have empty fields, and some games have fields in different formats. I wrote several parsers to handle these special cases.

Pitfall 3: Pagination

The pagination logic in the Bilibili Game Center is a bit strange. Sometimes requesting the first page returns duplicate data. I only solved this problem by adding deduplication logic.

Open Source

Project address: yfgug/bgame

If you’re interested in Bilibili game data, you can check it out. The code is average, but the functionality is complete.

Final Thoughts

I used this project to practice Python scraping. After finishing it, I realized that scraping isn’t technically difficult; the hard part is handling all kinds of edge cases.

Anti-scraping, data formats, pagination, encoding… every step can go wrong. Writing a scraper is like a battle of wits with the website’s developers. Sometimes you win, sometimes you lose.

But the data you finally scrape can indeed reveal some interesting things.

That’s probably the charm of data.

“Data is the oil of the new era, and analysis is the refinery.”