亚洲欧美日韩熟女|做爱高潮视频网址|国产一区二区三级片|国产Av中文字幕www.性色av|亚洲婷婷永久免费|国产高清中文字幕|欧美变态网站久re视频精品|人妻AV鲁丝第一页|天堂AV一区二区在线观看|综合 91在线精品

FastDFS高可用使用介紹

2023-04-12


FastDFS高可用

  • 目標
  • 安裝部署
  • 安裝依賴
  • 文件準備
  • 安裝步驟
  • 通用文件安裝
  • libfastcommon安裝（全節(jié)點安裝）
  • FastDFS_v5.05安裝（全節(jié)點安裝）
  • 單跟蹤節(jié)點單組存儲節(jié)點部分
  • 跟蹤器配置
  • 存儲器配置
  • 單組單節(jié)點配置
  • 單組多節(jié)點配置
  • 總結（t1-s1-s2）
  • 多跟蹤節(jié)點多組存儲節(jié)點部分
  • 跟蹤器配置
  • 存儲器配置
  • 測試
  • 安裝小結
  • nginx部署安裝
  • fastdfs-nginx-module模塊安裝
  • nginx安裝
  • nginx.conf配置
  • 多存儲節(jié)點安裝配置
  • 負載測試
  • 反向代理配置
  • 緩存設置
  • keepalived
  • nginx+keepalived+fastdfs
  • web項目集成
  • FastDFS資料
  • 基本模塊
  • 高可用
  • 參考資料
  • 日志分析
  • 存儲監(jiān)控

目標


實現(xiàn)統(tǒng)一的對外下載訪問入口的高可用架構，其中所有的Nginx只做下載用途。



機器

安裝說明

192.168.89.101

Libfastcommon，F(xiàn)astDFS

192.168.89.102

Libfastcommon，F(xiàn)astDFS

192.168.89.103

Libfastcommon，F(xiàn)astDFS，nginx，fastdfs-nginx-module

192.168.89.104

Libfastcommon，F(xiàn)astDFS，nginx，fastdfs-nginx-module

192.168.89.105

Libfastcommon，F(xiàn)astDFS，nginx，fastdfs-nginx-module

192.168.89.106

Libfastcommon，F(xiàn)astDFS，nginx，fastdfs-nginx-module

安裝部署


安裝依賴

[root@centos7-81 local]# yum -y install zlib zlib-devel pcre pcre-devel gcc gcc-c++ openssl openssl-devel libevent libevent-devel perl unzip

因為我本地虛擬機是新安裝的，所以都需要安裝一下，實際中缺哪項安裝哪項即可。


文件準備

文件名

說明

V1.0.7

libfastcommon-1.0.7（FastDFS全節(jié)點）

FastDFS_v5.05.tar.gz

（FastDFS全節(jié)點）

fastdfs-nginx-module_v1.16.tar.gz

解決組內同步延遲問題（存儲節(jié)點）

ngx_cache_purge-2.3.tar.gz

清除指定url的緩存

注：①本文檔中安裝版本都和此表格中版本對應。②安裝文件默認存放到主機的/usr/src目錄下。


安裝步驟

通用文件安裝

# 為了安裝簡便，先將防火墻關閉
[root@centos7-101 ~]# systemctl stop firewalld
# 先將文件上傳到101，全節(jié)點文件拷貝（102、103、104、105、106）
[root@centos7-101 src]# scp V1.0.7 192.168.89.102:/usr/src
[root@centos7-101 src]# scp FastDFS_v5.05.tar.gz 192.168.89.102:/usr/src
libfastcommon安裝（全節(jié)點安裝）
[root@centos7-101 src]# tar -zxvf V1.0.7 -C /usr/local
[root@centos7-101 src]# cd /usr/local/libfastcommon-1.0.7
[root@centos7-101 libfastcommon-1.0.7]# ./make.sh
[root@centos7-101 libfastcommon-1.0.7]# ./make.sh install

全節(jié)點完成安裝。


FastDFS_v5.05安裝（全節(jié)點安裝）
[root@centos7-101 src]# tar -zxvf FastDFS_v5.05.tar.gz -C /usr/local
[root@centos7-101 src]# cd /usr/local/FastDFS
[root@centos7-101 FastDFS]# ./make.sh
[root@centos7-101 FastDFS]# ./make.sh install

備注：命令執(zhí)行完，沒有錯誤日志且/etc/fdfs目錄下存在文件即安裝成功。


單跟蹤節(jié)點單組存儲節(jié)點部分

跟蹤器配置

以192.168.89.101節(jié)點為例進行如下說明。
（1）配置/etc/fdfs/tracker.conf，先拷貝一份，然后進行如下配置：


base_path=/fastdfs/tracker
# 配置存儲器
store_lookup=0

（2）創(chuàng)建base_path目錄


[root@centos7-101 fdfs]#  mkdir -p /fastdfs/tracker

（3）測試服務


# 啟動服務
[root@centos7-101 src]# fdfs_trackerd /etc/fdfs/tracker.conf 
[root@centos7-101 src]# ps -ef | grep fdfs
[root@centos7-101 src]# ss -tpln
存儲器配置
單組單節(jié)點配置

以192.168.89.103節(jié)點為例進行如下說明。
（1）配置/etc/fdfs/storage.conf，先拷貝一份（storage.conf.sample），然后進行如下配置。


group_name=group1
base_path=/fastdfs/storage
store_path0=/fastdfs/storage
tracker_server=192.168.89.101:22122

（2）創(chuàng)建base_path目錄


[root@centos7-103 fdfs]#  mkdir -p /fastdfs/storage

（3）測試【啟動服務】


# 啟動存儲服務
[root@centos7-103 fdfs]# fdfs_storaged /etc/fdfs/storage.conf
# 查看啟動日志
[root@centos7-103 ~]# tail -f /fastdfs/storage/logs/storaged.log
[2020-05-20 13:42:09] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-20 13:42:09] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-20 13:42:09] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.103, my_server_id_str: 192.168.89.103, g_server_id_in_filename: 1733929152
[2020-05-20 13:42:09] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.103  192.168.89.103
[2020-05-20 13:42:09] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.103
[2020-05-20 13:42:39] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122
[2020-05-20 13:42:09] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-20 13:42:09] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-20 13:42:09] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.103, my_server_id_str: 192.168.89.103, g_server_id_in_filename: 1733929152
[2020-05-20 13:42:09] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.103  192.168.89.103
[2020-05-20 13:42:09] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.103
[2020-05-20 13:42:39] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122

（4）測試【上傳照片】


# 修改控制器(192.168.89.101)客戶端配置文件
[root@centos7-101 fdfs]# vi client.conf 
# base_path=/fastdfs/tracker
# tracker_server=192.168.89.101:22122
# 上傳圖片
[root@centos7-101 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-1.jpg
group1/M00/00/00/wKhZZ17Ew1OAYOtBAAKLWRU1Cxw772.jpg
單組多節(jié)點配置

以192.168.89.103和192.168.89.105為例進行說明。
192.168.89.105節(jié)點配置按照103進行相同配置即可，配置完storage.conf文件后進行如下操作。
（1）啟動服務


# 啟動存儲服務
[root@centos7-103 fdfs]# fdfs_storaged /etc/fdfs/storage.conf
# 查看啟動日志
[root@centos7-103 ~]# tail -f /fastdfs/storage/logs/storaged.log
[2020-05-20 16:05:27] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-20 16:05:27] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-20 16:05:27] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.105, my_server_id_str: 192.168.89.105, g_server_id_in_filename: 1767483584
[2020-05-20 16:05:27] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.105  192.168.89.105
[2020-05-20 16:05:27] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.105
[2020-05-20 16:05:27] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122
[2020-05-20 16:05:27] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.103:23000

服務啟動后，在103上的圖片會自動同步過來。可以查看日志文件，出現(xiàn)如下（時間間隔）：
[2020-05-20 16:12:04] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.105:23000
（2）測試


# 上傳圖片
[root@centos7-101 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg 
group1/M00/00/00/wKhZZ17E5S2AMZf2AAKMU4U081o805.jpg
總結（t1-s1-s2）

由192.168.89.101【tracker-1】、192.168.89.103【group1-1】和192.168.89.105【group1-2】完成了單組多節(jié)點的搭建。
上傳測試：


[root@centos7-101 fdfs]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-2.jpg 
group1/M00/00/00/wKhZZ17EqGCAL9j8AAKZ_83sbhk602.jpg

日志分析：


# 105日志
[root@centos7-105 logs]# tail -f storaged.log 
mkdir data path: FD ...
mkdir data path: FE ...
mkdir data path: FF ...
data path: /fastdfs/storage/data, mkdir sub dir done.
[2020-05-20 11:38:53] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-20 11:38:53] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.105, my_server_id_str: 192.168.89.105, g_server_id_in_filename: 1767483584
[2020-05-20 11:38:53] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.105  192.168.89.105
[2020-05-20 11:38:53] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.105
[2020-05-20 11:38:53] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122
[2020-05-20 11:38:53] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.103:23000
# 103日志
[root@centos7-103 logs]# tail -f storaged.log
mkdir data path: FE ...
mkdir data path: FF ...
data path: /fastdfs/storage/data, mkdir sub dir done.
[2020-05-20 11:38:42] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-20 11:38:42] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.103, my_server_id_str: 192.168.89.103, g_server_id_in_filename: 1733929152
[2020-05-20 11:38:42] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.103  192.168.89.103
[2020-05-20 11:38:42] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.103
[2020-05-20 11:39:12] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122
[2020-05-20 11:39:12] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.105:23000
[2020-05-20 11:39:42] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.105:23000

多跟蹤節(jié)點多組存儲節(jié)點部分

跟蹤器配置

以192.168.89.101和192.168.89.102為例進行說明。安裝同“單跟蹤節(jié)點單組存儲節(jié)點部分跟蹤器配置部分”。


存儲器配置
  • 【192.168.89.103】和【192.168.89.105】

該兩個節(jié)點都屬于group1，修改/etc/fdfs/storage.conf配置如下：


group_name=group1
base_path=/fastdfs/storage
store_path0=/fastdfs/storage
tracker_server=192.168.89.101:22122
tracker_server=192.168.89.102:22122
  • 【192.168.89.104】和【192.168.89.106】

該兩個節(jié)點都屬于group2，修改/etc/fdfs/storage.conf配置如下：


group_name=group2
base_path=/fastdfs/storage
store_path0=/fastdfs/storage
tracker_server=192.168.89.101:22122
tracker_server=192.168.89.102:22122
測試

在192.168.89.102（T2）跟蹤器上傳一個圖片，可以看到跟蹤器的輪詢效果，如下所示：


[root@centos7-102 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg 
group1/M00/00/00/wKhZZ17E9TmADDyNAAKMU4U081o921.jpg
[root@centos7-102 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg 
group2/M00/00/00/wKhZal7E9UKAeIPjAAKMU4U081o129.jpg
[root@centos7-102 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg 
group1/M00/00/00/wKhZaV7E9UiARRYYAAKMU4U081o553.jpg
[root@centos7-102 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg 
group2/M00/00/00/wKhZaF7E9UqAM212AAKMU4U081o334.jpg

安裝小結

以上部分完成了FastDFS的安裝，已經實現(xiàn)上傳文件。為了實現(xiàn)圖片展示的高可用，我們還需配置ngx_fastdfs_module的配置。后面章節(jié)進行插件的說明。


nginx部署安裝

nginx作用：存儲節(jié)點（103、104、105、106）做靜態(tài)資源代理；非存儲節(jié)點（107、108）做反向代理。


fastdfs-nginx-module模塊安裝

凡是存儲節(jié)點都需要安裝此插件，此案例中103、104、105、106節(jié)點都需要安裝。
（1）上傳、解壓fastdfs-nginx-module_v1.16.tar.gz


[root@centos7-103 src]# tar -zxvf fastdfs-nginx-module_v1.16.tar.gz -C /usr/local/

（2）修改config文件


[root@centos7-103 src]# cd /usr/local/fastdfs-nginx-module/src
[root@centos7-103 src]# vi config       # 去掉路徑中的local
CORE_INCS="$CORE_INCS /usr/local/include/fastdfs /usr/local/include/fastcommon/"

（3）配置mod_fastdfs.conf文件


[root@centos7-103 src]# cp /usr/local/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs
[root@centos7-103 src]# vi /etc/fdfs/mod_fastdfs.conf
base_path=/fastdfs/storage
tracker_server=192.168.89.101:22122  # 單跟蹤節(jié)點
tracker_server=192.168.89.102:22122  # 多跟蹤節(jié)點
group_name=group1 # 屬于哪個分組存儲節(jié)點，就配置哪個名稱
url_have_group_name = true
store_path0=/fastdfs/storage # 當前節(jié)點的實際存儲位置
group_count = 2  # 0表示單個組，一個以上按照實際分租數(shù)量進行配置
[group1]         # 多組才進行如下配置
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/fastdfs/storage
[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/fastdfs/storage

（4）拷貝http.conf、mime.types


[root@centos7-103 src]# cd /usr/local/FastDFS/conf
[root@centos7-103 conf]# cp http.conf mime.types /etc/fdfs

（5）配置軟連接


[root@centos7-103 src]# ln -s /fastdfs/storage/data /fastdfs/storage/data/M00

至此Nginx擴展模塊已安裝完畢。


nginx安裝

（1）安裝


[root@centos7-103 src]# tar -zxvf nginx-1.16.1.tar.gz -C /usr/local
[root@centos7-103 src]# cd /usr/local/nginx-1.16.1/
[root@centos7-103 nginx-1.16.1]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/fastdfs-nginx-module/src
[root@centos7-103 nginx-1.16.1]# make && make install

（2）啟動服務


[root@centos7-103 nginx-1.16.1]# /usr/local/nginx/sbin/nginx

（3）瀏覽器測試，輸入地址：http://192.168.89.103/
103機器命令行輸入：[root@centos7-102 src]# curl http://192.168.89.3
顯示“Welcome to nginx”等字樣，說明服務啟動成功。


nginx.conf配置
location /group1/M00 {
        ngx_fastdfs_module;
    }

在瀏覽器測試：http://192.168.89.103/group1/M00/00/00/wKhZZ17Ew1OAYOtBAAKLWRU1Cxw772.jpg


多存儲節(jié)點安裝配置

（1）mod_fastdfs.conf文件【103、104、105、106】


base_path=/fastdfs/storage
tracker_server=192.168.89.101:22122
tracker_server=192.168.89.102:22122
group_name=group1 # 103 105 為group1，104 106為group2
url_have_group_name = true
store_path0=/fastdfs/storage # 當前節(jié)點的實際存儲位置
group_count = 2  # 此案例共2組
[group1]         # 多組配置
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/fastdfs/storage
[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/fastdfs/storage

（2）nginx.conf配置


location ~/group([1-2])/M00 {
        ngx_fastdfs_module;
    }

其他按照單節(jié)點的安裝即可（上面三節(jié)部分）。


負載測試

（1）啟動服務


[root@centos7-101 src]# fdfs_trackerd /etc/fdfs/tracker.conf
[root@centos7-102 src]# fdfs_trackerd /etc/fdfs/tracker.conf
[root@centos7-103 src]# fdfs_storaged /etc/fdfs/storage.conf
[root@centos7-104 src]# fdfs_storaged /etc/fdfs/storage.conf
[root@centos7-105 src]# fdfs_storaged /etc/fdfs/storage.conf
[root@centos7-106 src]# fdfs_storaged /etc/fdfs/storage.conf

（2）日志分析


[root@centos7-101 src]# tail -f /fastdfs/tracker/logs/trackerd.log 
[2020-05-22 10:07:42] INFO - FastDFS v5.05, base_path=/fastdfs/tracker, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=22122, bind_addr=, max_connections=256, accept_threads=1, work_threads=4, store_lookup=0, store_group=, store_server=0, store_path=0, reserved_storage_space=10.00%, download_server=0, allow_ip_count=-1, sync_log_buff_interval=10s, check_active_interval=120s, thread_stack_size=64 KB, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, use_storage_id=0, id_type_in_filename=ip, storage_id_count=0, rotate_error_log=0, error_log_rotate_time=00:00, rotate_error_log_size=0, log_file_keep_days=0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-22 10:07:42] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.101  192.168.89.101
[2020-05-22 10:08:08] INFO - file: tracker_relationship.c, line: 383, selecting leader...
[2020-05-22 10:08:09] INFO - file: tracker_service.c, line: 969, the tracker leader is 192.168.89.102:22122
[root@centos7-102 src]# tail -f /fastdfs/tracker/logs/trackerd.log 
[2020-05-22 10:07:51] INFO - FastDFS v5.05, base_path=/fastdfs/tracker, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=22122, bind_addr=, max_connections=256, accept_threads=1, work_threads=4, store_lookup=0, store_group=, store_server=0, store_path=0, reserved_storage_space=10.00%, download_server=0, allow_ip_count=-1, sync_log_buff_interval=10s, check_active_interval=120s, thread_stack_size=64 KB, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, use_storage_id=0, id_type_in_filename=ip, storage_id_count=0, rotate_error_log=0, error_log_rotate_time=00:00, rotate_error_log_size=0, log_file_keep_days=0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-22 10:07:51] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.102  192.168.89.102
[2020-05-22 10:08:10] INFO - file: tracker_relationship.c, line: 383, selecting leader...
[2020-05-22 10:08:10] INFO - file: tracker_relationship.c, line: 401, I am the new tracker leader 192.168.89.102:22122
[root@centos7-103 src]# tail -f /fastdfs/storage/logs/storaged.log 
[2020-05-22 10:08:33] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-22 10:08:33] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-22 10:08:33] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.103, my_server_id_str: 192.168.89.103, g_server_id_in_filename: 1733929152
[2020-05-22 10:08:33] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.103  192.168.89.103
[2020-05-22 10:08:33] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.102:22122, as a tracker client, my ip is 192.168.89.103
[2020-05-22 10:08:33] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.103
[2020-05-22 10:08:33] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.102:22122, set tracker leader: 192.168.89.102:22122
[2020-05-22 10:09:04] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.105:23000
[root@centos7-104 nginx-1.16.1]# tail -f /fastdfs/storage/logs/storaged.log 
[2020-05-22 10:08:09] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group2, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-22 10:08:09] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-22 10:08:09] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.104, my_server_id_str: 192.168.89.104, g_server_id_in_filename: 1750706368
[2020-05-22 10:08:09] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.104  192.168.89.104
[2020-05-22 10:08:09] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.102:22122, as a tracker client, my ip is 192.168.89.104
[2020-05-22 10:08:09] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.104
[2020-05-22 10:08:39] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.102:22122, set tracker leader: 192.168.89.102:22122
[2020-05-22 10:09:10] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.106:23000
[root@centos7-105 src]# tail -f /fastdfs/storage/logs/storaged.log 
[2020-05-22 10:08:35] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-22 10:08:35] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-22 10:08:35] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.105, my_server_id_str: 192.168.89.105, g_server_id_in_filename: 1767483584
[2020-05-22 10:08:36] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.105  192.168.89.105
[2020-05-22 10:08:36] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.105
[2020-05-22 10:08:36] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.102:22122, as a tracker client, my ip is 192.168.89.105
[2020-05-22 10:08:36] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.102:22122, set tracker leader: 192.168.89.102:22122
[2020-05-22 10:08:36] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.103:23000
[root@centos7-106 nginx-1.16.1]# tail -f /fastdfs/storage/logs/storaged.log 
[2020-05-22 10:08:40] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group2, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2020-05-22 10:08:40] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2020-05-22 10:08:40] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.106, my_server_id_str: 192.168.89.106, g_server_id_in_filename: 1784260800
[2020-05-22 10:08:40] INFO - local_host_ip_count: 3,  127.0.0.1  10.0.2.106  192.168.89.106
[2020-05-22 10:08:40] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.102:22122, as a tracker client, my ip is 192.168.89.106
[2020-05-22 10:08:40] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.106
[2020-05-22 10:08:40] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.102:22122, set tracker leader: 192.168.89.102:22122
[2020-05-22 10:08:40] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.104:23000

（3）上傳測試


[root@centos7-101 fdfs]# vi client.conf
[root@centos7-101 fdfs]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-2.jpg 
group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg
[root@centos7-101 fdfs]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg 
group2/M00/00/00/wKhZaF7HNmqABRnyAAKMU4U081o247.jpg

（4）訪問測試
http://192.168.89.103:8000/group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg http://192.168.89.104:8000/group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg
按照道理104應該訪問不到，因為group1是在103、105上存儲，可實際結果顯示沒問題。經過測試：在四個存儲節(jié)點否正常的情況下，使用103、105訪問group2的存儲，重定向到104；使用104、106訪問group1的存儲，重定向到103。日志如下：


[root@centos7-104 ~]# tail -f /usr/local/nginx/logs/error.log
2020/05/22 10:21:29 [error] 5189#0: *1 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.89.1, server: localhost, request: "GET /group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg HTTP/1.1", upstream: "http://192.168.89.103:8000/group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg?redirect=1", host: "192.168.89.104:8000"

從日志中可以看出，重定向到了103的服務。


反向代理配置

我們使用192.168.89.107、192.168.89.108倆節(jié)點進行反向代理配置。
（1）安裝nginx


[root@centos7-107 src]# tar -zxvf /usr/src/nginx-1.16.1.tar.gz -C /usr/local
[root@centos7-107 src]# cd /usr/local/nginx-1.16.1
[root@centos7-107 src]# ./configure --prefix=/usr/local/nginx
[root@centos7-107 src]# cd /usr/local/nginx-1.16.1

（2）/usr/local/nginx/conf/nginx.conf配置


upstream fdfs_group1 {
    server 192.168.89.103:8000 weight=1 max_fails=2 fail_timeout=30s;
    server 192.168.89.105:8000 weight=1 max_fails=2 fail_timeout=30s;
}

upstream fdfs_group2 {
    server 192.168.89.104:8000 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.89.106:8000 weight=1 max_fails=2 fail_timeout=30s;
}
location /group1/M00 {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_pass http://fdfs_group1;
expires 30d;
}
location /group2/M00 {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_pass http://fdfs_group2;
expires 30d;
}

（3）測試
http://192.168.89.107:8081/group2/M00/00/00/wKhZaF7HNmqABRnyAAKMU4U081o247.jpg http://192.168.89.108:8081/group2/M00/00/00/wKhZaF7HNmqABRnyAAKMU4U081o247.jpg
nginx.conf參考配置：


server_names_hash_bucket_size 128;
client_header_buffer_size 32k;
large_client_header_buffers 4 32k;
client_max_body_size 300m;

proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 16k;
proxy_buffers 4 64k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 128k;

proxy_cache_path /fastdfs/cache/nginx/proxy_cache levels=1:2 keys_zone=http-cache:200m max_size=1g inactive=30d;
proxy_temp_path /fastdfs/cache/nginx/proxy_cache/tmp;

upstream fdfs_group1 {
    server 192.168.89.106:8888 weight=1 max_fails=2 fail_timeout=30s;
    server 192.168.89.108:8888 weight=1 max_fails=2 fail_timeout=30s;
}

upstream fdfs_group2 {
    server 192.168.89.105:8888 weight=1 max_fails=2 fail_timeout=30s;
    server 192.168.89.107:8888 weight=1 max_fails=2 fail_timeout=30s;
}

location /group1/M00 {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_cache http-cache;
proxy_cache_valid 200 304 12h;
proxy_cache_key $uri$is_args$args;
proxy_pass http://fdfs_group1;
expires 30d;
}
location /group2/M00 {
#proxy_next_upstream http_502 http_504 error timeout invalid_header;
#proxy_cache http-cache;
#proxy_cache_valid 200 304 12h;
#proxy_cache_key $uri$is_args$args;
proxy_pass http://fdfs_group2;
expires 30d;
}
#清除緩存的訪問權限
location ~/purge(/.*) {
allow 127.0.0.1;
allow 192.168.89.0/24;
deny all;
proxy_cache_purge http-cache $1$is_args$args;
}
緩存設置

命令：


[root@centos7-81 src]# tar -zxvf ngx_cache_purge-2.3.tar.gz -C /usr/local
[root@centos7-83 nginx-1.16.1]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/ngx_cache_purge-2.3
[root@centos7-83 nginx-1.16.1]# make && make install
mkdir –p /fastdfs/cache/nginx/proxy_cache
mkdir –p /fastdfs/cache/nginx/proxy_cache/tmp

測試：
http://192.168.89.3:8000/group2/M00/00/00/wKhZBl7CNgaAQuHFAAKZ_83sbhk190.jpg http://192.168.89.4:8000/group2/M00/00/00/wKhZBl7CNgaAQuHFAAKZ_83sbhk190.jpg


keepalived

我們使用192.168.89.109、192.168.89.110倆節(jié)點進行keepalived配置。
（1）安裝nginx
（2）安裝keepalived


[root@centos7-109 src]# tar -zxvf keepalived-2.0.20.tar.gz
[root@centos7-109 src]# cd keepalived-2.0.20
[root@centos7-109 keepalived-2.0.20]# ./configure --prefix=/usr/local/keepalived
[root@centos7-109 keepalived-2.0.20]# make && make install

完成后，在生成路徑和文件如下：
/usr/local/keepalived/etc/keepalived/keepalived.conf
/usr/local/keepalived/etc/sysconfig/keepalived
/usr/local/keepalived/sbin/keepalived
（3）初始化及啟動


# keepalived啟動腳本變量引用文件，默認路徑是/etc/sysconfig
[root@centos7-109 src]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig
# 將keepalived主程序加入到環(huán)境變量
[root@centos7-109 sbin]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin
[root@centos7-109 sbin]# cp /usr/src/keepalived-2.0.20/keepalived/etc/init.d/keepalived /etc/init.d/
# 將配置文件放到默認路徑下
[root@centos7-109 etc]# mkdir /etc/keepalived
[root@centos7-109 etc]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

（4）192.168.89.109節(jié)點配置


# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id LVS_109
}

vrrp_script chk_nginx {
    script "/etc/keepalived/check_list"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface enp0s8
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
}
# 將track_script塊加入instance配置快
track_script {
    chk_nginx # 執(zhí)行Nginx監(jiān)控服務
}
    virtual_ipaddress {
        192.168.89.100
    }
}

（5）192.168.89.110節(jié)點配置


# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id LVS_110
}

vrrp_script chk_nginx {
    script "/etc/keepalived/check_list"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s8
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
}
# 將track_script塊加入instance配置快
track_script {
    chk_nginx # 執(zhí)行Nginx監(jiān)控服務
}
    virtual_ipaddress {
        192.168.89.100
    }
}

（6）check_list腳本
用于檢測nginx存活狀態(tài)的腳本，如果nginx不存活，則關閉宕掉nginx主機上的keepalived。
因為在keepalived+nginx架構中，nginx宕機后，用戶請求會失敗，但是keepalived不會進行切換。


#!/bin/sh

nginxpid=$(ps -C nginx --no-header|wc -l)
#1.判斷Nginx是否存活,如果不存活則嘗試啟動Nginx
if [ $nginxpid -eq 0 ];then
    systemctl start nginx
    sleep 3
    #2.等待3秒后再次獲取一次Nginx狀態(tài)
    nginxpid=$(ps -C nginx --no-header|wc -l) 
    #3.再次進行判斷, 如Nginx還不存活則停止Keepalived,讓地址進行漂移,并退出腳本  
    if [ $nginxpid -eq 0 ];then
        systemctl stop keepalived
   fi
fi

（7）測試
瀏覽器中輸入：http://192.168.89.100/
顯示109節(jié)點的頁面，然后停掉109節(jié)點的nginx和keepalived服務，在訪問可以看到地址進行了漂移，顯示110節(jié)點的頁面。


nginx+keepalived+fastdfs

在《keepalived》章節(jié)我們進行了nginx服務高可用的配置，本章節(jié)我們將fastdfs的代理加入進來。使用109，110環(huán)境進行配置說明。
（1）nginx配置【192.168.89.109和192.168.89.110】


# vi /usr/local/nginx/conf/nginx.conf
upstream fdfs_group1 {  # 添加上游服務器
        server 192.168.89.103:8000 weight=1 max_fails=2 fail_timeout=30s;
        server 192.168.89.105:8000 weight=1 max_fails=2 fail_timeout=30s;
    }
    upstream fdfs_group2 {  # 添加上游服務器
        server 192.168.89.104:8000 weight=1 max_fails=2 fail_timeout=30s;
        server 192.168.89.106:8000 weight=1 max_fails=2 fail_timeout=30s;
}
server {
        listen       80;
        server_name  localhost;
        …………
        location /group1/M00 {  # 反向代理
            proxy_pass http://fdfs_group1;
            expires 30d;
        }
        location /group2/M00 {
            proxy_pass http://fdfs_group2;
            expires 30d;
        }
…………

先修改109的配置，然后同步到110，[root@centos7-109 conf]# scp nginx.conf 192.168.89.110:/usr/local/nginx/conf。
（2）啟動服務
101、102啟動fastdfs tracker服務。
103、104、105、106啟動fastdfs storage和nginx服務。
109、110啟動keepalived和nginx服務。


# 101 102 tracker服務啟動
[root@centos7-101 ~]# fdfs_trackerd /etc/fdfs/tracker.conf
[root@centos7-102 ~]# fdfs_trackerd /etc/fdfs/tracker.conf
# 103 104 105 105 storage、nginx服務啟動
[root@centos7-103 ~]# fdfs_storaged /etc/fdfs/storage.conf
[root@centos7-103 ~]# /usr/local/nginx/sbin/nginx
……
# 109 110 keepalived、nginx服務啟動

（3）測試
服務都正常啟動后，我們進行測試。
首先在101或者102服務器上傳一個圖片，如下所示：


# 101 上傳圖片
[root@centos7-101 src]# fdfs_upload_file /etc/fdfs/client.conf t-1.jpg 
group1/M00/00/00/wKhZZ17OEOCAMRH1AAKLWRU1Cxw036.jpg
[root@centos7-101 src]# fdfs_upload_file /etc/fdfs/client.conf t-2.jpg 
group2/M00/00/00/wKhZaF7OETOAcwfhAAKZ_83sbhk275.jpg

使用瀏覽器訪問如下地址：
http://192.168.89.100/group1/M00/00/00/wKhZZ17OEOCAMRH1AAKLWRU1Cxw036.jpg http://192.168.89.100/group2/M00/00/00/wKhZaF7OETOAcwfhAAKZ_83sbhk275.jpg


web項目集成

（1）依賴jar：fastdfs_client.jar
（2）初始化客戶端核心代碼，寫到靜態(tài)塊或者構造器中：


# class根文件目錄
String classPath = 
new File(ImgUploadService.class.getResource("/").getFile()).getCanonicalPath();
	// 根據fdfs_client.conf
ClientGlobal.init(classPath + File.separator + configFile);
	trackerClient = new TrackerClient();
	trackerServer = trackerClient.getConnection();
	storageServer = null;
	storageClient = new StorageClient(trackerServer, storageServer);
	log.info("init: " + storageClient.hashCode());
	// 給dfs發(fā)送一個消息，否則第一次傳輸會有異常信息
    ProtoCommon.activeTest(trackerServer.getSocket());

（3）上傳核心代碼


TrackerServer trackerServer = trackerClient.getConnection();
StorageClient storageClient = new StorageClient(trackerServer, storageServer);
// results數(shù)組中第一個元素為存儲組名，第二個元素為圖片存儲路徑。
	String[] results = storageClient.upload_file(fileContent, suffix, null);

（4）下載核心代碼


public byte[] getFileByteBuffer(Efilesummary object) throws Exception {
		String classPath = new File(UploadService.class.getResource("/").getFile()).getCanonicalPath();
		ClientGlobal.init(classPath + File.separator + configFile);

	    TrackerClient trackerClient = new TrackerClient();
	    TrackerServer trackerServer = trackerClient.getConnection();
	    StorageServer storageServer = null;
	    StorageClient storageClient = new StorageClient(trackerServer, storageServer);

	    return storageClient.download_file(object.getPath(), object.getRemotefsid());
	}

（5）刪除核心代碼


public void deleteFile(Efilesummary object) throws Exception {
		String classPath = new File(UploadService.class.getResource("/").getFile()).getCanonicalPath();
		ClientGlobal.init(classPath + File.separator + configFile);

	    TrackerClient trackerClient = new TrackerClient();
	    TrackerServer trackerServer = trackerClient.getConnection();
	    StorageServer storageServer = null;
	    StorageClient storageClient = new StorageClient(trackerServer, storageServer);

	    int i = storageClient.delete_file(object.getPath(), object.getRemotefsid());
	    System.out.println(i);
	}

（6）demo
fastdfs-demo.rar


FastDFS資料


基本模塊



Tracker：跟蹤控制器，程序中會使用該服務器的IP地址。


Storage：存儲器，組和組之間代表擴容關系，組內之間塊與塊代表備份關系。


高可用


參考資料

https://blog.51cto.com/7072753/2286604



ip route del default via 192.168.89.1 這個命令加到 ~/.bashrc


  • keepalived資料：https://www.jianshu.com/p/a6b5ab36292a

日志分析


存儲監(jiān)控


[root@centos7-104 fdfs]# fdfs_monitor /etc/fdfs/storage.conf
[2020-05-20 17:10:26] DEBUG - base_path=/fastdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=2, server_index=1

tracker server is 192.168.89.102:22122

group count: 2

Group 1:
group name = group1
disk total space = 8662 MB
disk free space = 7544 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

	Storage 1:
		id = 192.168.89.103
		ip_addr = 192.168.89.103  ACTIVE
		http domain = 
		version = 5.05
		join time = 2020-05-20 11:38:40
		up time = 2020-05-20 17:05:32
		total storage = 8662 MB
		free storage = 7544 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		source storage id = 
		if_trunk_server = 0
		connection.alloc_count = 256
		connection.current_count = 1
		connection.max_count = 1
		total_upload_count = 4
		success_upload_count = 4
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 670980
		success_upload_bytes = 670980
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 4
		success_file_open_count = 4
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 4
		success_file_write_count = 4
		last_heart_beat_time = 2020-05-20 17:10:01
		last_source_update = 2020-05-20 16:11:23
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 
	Storage 2:
		id = 192.168.89.105
		ip_addr = 192.168.89.105  ACTIVE
		http domain = 
		version = 5.05
		join time = 2020-05-20 11:38:50
		up time = 2020-05-20 17:06:32
		total storage = 8662 MB
		free storage = 7544 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		source storage id = 192.168.89.103
		if_trunk_server = 0
		connection.alloc_count = 256
		connection.current_count = 1
		connection.max_count = 1
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 670980
		success_sync_in_bytes = 670980
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 4
		success_file_open_count = 4
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 4
		success_file_write_count = 4
		last_heart_beat_time = 2020-05-20 17:10:03
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 2020-05-20 16:12:03
		last_synced_timestamp = 1970-01-01 08:00:00 (never synced)

Group 2:
group name = group2
disk total space = 8662 MB
disk free space = 7545 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 1
current trunk file id = 0

	Storage 1:
		id = 192.168.89.104
		ip_addr = 192.168.89.104  ACTIVE
		http domain = 
		version = 5.05
		join time = 2020-05-20 16:43:25
		up time = 2020-05-20 17:05:54
		total storage = 8662 MB
		free storage = 7545 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		source storage id = 
		if_trunk_server = 0
		connection.alloc_count = 256
		connection.current_count = 1
		connection.max_count = 1
		total_upload_count = 1
		success_upload_count = 1
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 166995
		success_upload_bytes = 166995
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 1
		success_file_open_count = 1
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 1
		success_file_write_count = 1
		last_heart_beat_time = 2020-05-20 17:10:23
		last_source_update = 2020-05-20 16:47:08
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 
	Storage 2:
		id = 192.168.89.106
		ip_addr = 192.168.89.106  ACTIVE
		http domain = 
		version = 5.05
		join time = 2020-05-20 16:43:33
		up time = 2020-05-20 17:07:02
		total storage = 8662 MB
		free storage = 7545 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		source storage id = 192.168.89.104
		if_trunk_server = 0
		connection.alloc_count = 256
		connection.current_count = 1
		connection.max_count = 1
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 166995
		success_sync_in_bytes = 166995
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 1
		success_file_open_count = 1
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 1
		success_file_write_count = 1
		last_heart_beat_time = 2020-05-20 17:10:01
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 2020-05-20 16:47:17
		last_synced_timestamp = 2020-05-20 16:47:08 (0s delay)



本文僅代表作者觀點，版權歸原創(chuàng)者所有，如需轉載請在文中注明來源及作者名字。

免責聲明：本文系轉載編輯文章，僅作分享之用。如分享內容、圖片侵犯到您的版權或非授權發(fā)布，請及時與我們聯(lián)系進行審核處理或刪除，您可以發(fā)送材料至郵箱：service@tojoy.com