Compare commits

...

37 Commits

Author SHA1 Message Date
ygxbnet b100fc9b32 fix(upload): 修正上传文件清空确认对话框文本
构建上传工具 / build (push) Successful in 1m51s
- 修正了标题文本"是否确认清空需要上传文件"
- 修正了提示文本中的"tmp"文件夹显示格式
- 更新了确认对话框的内容描述文本
2026-05-01 22:51:12 +08:00
ygxbnet d426c16104 refactor(uploader): 优化上传器代码结构和上下文取消处理
- 在进度更新循环中添加上下文取消检查点
- 在文件复制操作前添加上下文取消检查点
- 重构代码缩进和括号位置以提高可读性
- 优化 goroutine 中的上下文取消处理逻辑
- 统一代码块的括号格式和缩进风格
2026-05-01 22:47:39 +08:00
ygxbnet 993814cdfa refactor(uploader): 优化文件信息统计逻辑
- 将变量名 fInfo 重命名为 filesInfo 以提高可读性
- 调整代码顺序,将 AddLog 调用移到变量声明后
- 统一使用新变量名在所有相关位置进行引用
- 移动 g.SetLimit 注释位置以提高代码可读性
2026-05-01 22:37:45 +08:00
ygxbnet 8ffa0531d6 feat(uploader): 添加文件上传配置和临时文件处理功能
构建上传工具 / build (push) Successful in 1m10s
- 新增 ClearFilesNoPrompt 配置项用于控制是否提示清空文件
- 实现临时文件目录(tmp)管理,优先处理未上传完的文件
- 添加文件复制功能,支持快速零拷贝技术
- 实现文件清空提示机制,支持用户确认操作
- 优化文件上传流程,添加进度跟踪和状态更新
- 过滤掉大小为0的文件,避免无效上传
- 修改数据结构名称提升代码可读性
2026-05-01 21:01:01 +08:00
ygxbnet 3f6e999783 feat(app): 添加文件清空确认对话框功能
- 引入 ConfirmClearDialog 组件用于清空文件确认
- 添加 clear-files-no-prompt 配置项控制是否显示确认弹窗
- 实现清空文件列表显示和确认逻辑
- 集成 Element Plus 图标组件库
- 优化日志输出格式增加空格分隔
- 重构配置写入方法使用统一的 configModel 枚举
- 添加事件监听处理清空文件操作
- 实现勾选不再提示选项并保存配置
2026-05-01 20:45:17 +08:00
ygxbnet 03e4e6f45b fix(uploader): 修复上传器逻辑和UI提示问题
构建上传工具 / build (push) Successful in 1m23s
- 修复目录选择为空时不更改配置的逻辑
- 将警告提示改为错误提示以提高用户体验
- 优化停止运行时的成功提示
- 增加进度列表的最大高度以显示更多内容
- 调整定时器逻辑顺序避免提前退出
- 移除重复的任务等待错误处理
- 优化代码结构和空行格式
2026-04-30 21:24:30 +08:00
ygxbnet 6bd82024d9 refactor(App): 移除服务器地址和自动启动配置功能
- 注释掉 serverUrl 相关的响应式变量定义
- 注释掉 autoStart 相关的响应式变量定义
- 移除进度列表中的测试数据注释
- 注释掉服务器地址验证逻辑
- 注释掉服务器地址和自动启动的写入配置函数
- 注释掉配置加载中的相关字段赋值
- 在模板中隐藏服务器地址输入框和自动启动复选框组件
2026-04-30 20:39:08 +08:00
ygxbnet 6952c33f16 refactor(ci): 重命名构建工作流文件
构建上传工具 / build (push) Successful in 2m56s
- 将 .gitea/workflows/build_tool.yaml 重命名为 .gitea/workflows/build.yaml
- 更新工作流中的作业名称从 build-tool 到 build
- 保持相同的触发条件和运行环境配置
2026-04-29 14:07:22 +08:00
ygxbnet 3d2b3469cc refactor(uploader): 重命名文件上传函数并优化代码结构
构建上传工具 / build-tool (push) Successful in 1m59s
- 将 StartLooking 函数重命名为 StartUpload 以更准确反映功能
- 修复构建脚本中的版本标记 ldflags 格式问题
- 更新应用标题格式,在版本号前添加冒号分隔符
- 优化上传进度循环逻辑,调整代码执行顺序
- 添加注释说明上传程序启动和文件处理功能
- 清理代码中的多余空行和变量声明
2026-04-29 14:01:50 +08:00
ygxbnet a528d6a877 fix(uploader): 修复上传器上下文取消时的资源泄露
构建上传工具 / build-tool (push) Successful in 1m19s
- 在上下文取消时正确关闭 lines channel
- 防止 goroutine 阻塞导致的内存泄露
- 确保所有资源在程序退出时正确释放
2026-04-28 15:44:48 +08:00
ygxbnet 73a7d26816 refactor(uploader): 优化文件上传并发处理逻辑
构建上传工具 / build-tool (push) Successful in 1m46s
- 将初始化连接池的并发方式改为waitgroup控制的goroutine池
- 调整文件处理时的channel缓冲区大小从100增加到200
- 移除不必要的sync.WaitGroup变量声明
- 修改进度计算逻辑,确保上传完成时进度显示为100%
- 添加对processLines函数的功能注释
- 优化上下文取消时的资源清理流程,及时关闭channel
2026-04-28 15:43:02 +08:00
ygxbnet 1cac9e9013 fix(build): 修复版本号链接参数格式问题
- 修正了 ldflags 中版本号格式导致的编译错误
- 移除了版本字符串中的意外空格
- 确保版本号正确传递给主程序包变量
2026-04-28 15:08:21 +08:00
ygxbnet 7f0e4fe607 perf(api): 提高并发请求限制并添加配置写入日志
构建上传工具 / build-tool (push) Successful in 1m40s
- 将并发请求限制从10提升到500
- 在配置写入时添加调试日志输出
2026-04-28 15:03:13 +08:00
ygxbnet 199bd43b00 feat(api): 优化HTTP连接池和并发控制
- 增加IdleConnTimeout从30秒到30分钟
- 添加并发请求限制通道,最大同时请求数为10
- 实现InitConn函数用于预创建连接池
- 在UploadDataToServer中添加请求限流控制
- 优化资源清理逻辑,使用defer确保响应体关闭
- 重命名runtime包别名以避免冲突
- 在uploader中添加连接池初始化日志
- 添加panic恢复机制和错误处理
2026-04-28 15:00:15 +08:00
ygxbnet 4addc29b2c refactor(config): 添加配置写入的并发安全锁机制
- 引入 sync.Mutex 确保配置访问的线程安全性
- 在 WriteConfig 函数中实现读写锁定机制
- 防止多协程同时修改配置导致的数据竞争问题
2026-04-28 14:59:21 +08:00
ygxbnet 7face117f3 refactor(App): 解决程序运行时重复写入配置
- 移除 watch 监听器,改用事件驱动方式保存配置
- 添加 writeServerUrl、writeToken 等配置保存方法
- 在表单项上绑定 change 事件触发配置保存
- 移除不再使用的 nextTick 导入
- 统一配置保存逻辑到独立函数中
2026-04-28 14:53:19 +08:00
ygxbnet 602c4c8546 fix(uploader): 修复上传进度初始化逻辑
构建上传工具 / build-tool (push) Successful in 1m1s
- 在处理文件前先清除之前的进度记录
- 将循环变量名从 k, v 更改为 fileName, lines 提高可读性
- 移动进度初始化位置确保每个文件都有正确的进度跟踪
- 删除重复的进度清除操作避免潜在的数据丢失问题
2026-04-28 01:35:16 +08:00
ygxbnet 75de353af6 refactor(uploader): 优化文件上传处理逻辑和资源管理
构建上传工具 / build-tool (push) Successful in 1m20s
- 简化响应体关闭逻辑,移除不必要的nil检查
- 调整后台状态推送频率,从500ms改为250ms
- 修复前端事件监听器注册顺序
- 移除未使用的进度变量
- 优化goroutine中的任务执行逻辑
- 改进文件路径显示,统一使用文件名而非完整路径
- 添加waitgroup等待确保资源正确释放
2026-04-28 01:07:43 +08:00
ygxbnet 12ef425b01 refactor(app): 移除调试打印语句
- 删除了 WriteConfig 方法中的 fmt.Println 调试代码
- 保留了核心配置写入功能不变
2026-04-28 00:14:06 +08:00
ygxbnet 34a3a70569 style(frontend): 调整表单项间距样式并优化进度列表布局
- 移除表单项内部的 8px 间距
- 为进度列表添加 12px 上边距以改善视觉层次
- 在上传器初始化时清除进度状态确保界面一致性
2026-04-28 00:01:54 +08:00
ygxbnet b050c36904 feat(App): 添加上传进度列表排序功能
- 引入 computed 属性用于对进度列表进行排序
- 按照上传状态优先级排序:已完成 > 未开始 > 上传中
- 将排序后的列表绑定到模板中的进度显示组件
- 优化用户体验,让完成和未开始的文件更易识别
2026-04-27 23:50:40 +08:00
ygxbnet f96f23360c feat(app): 添加自动启动和日志滚动功能并优化上传逻辑
- 增加了运行时自动启动上传配置选项
- 实现了日志输出的滚动控制功能
- 优化了上传进度显示和状态同步机制
- 提升了HTTP客户端连接池配置至500
- 改进了文件上传完成后的清理逻辑
- 添加了上下文取消检查避免资源泄露
- 完善了上传开始时的日志信息输出
2026-04-27 23:40:10 +08:00
ygxbnet d4cc335fbf refactor(app): 重构应用状态管理和配置常量定义
- 将全局变量 isRun 移动到 App 结构体内部作为实例字段
- 在 config.go 中定义配置键名为常量,提高代码可维护性
- 使用结构体实例字段替代全局变量管理上传状态
- 修改 StartLooking 函数中的上下文取消处理逻辑
- 移除上传程序退出日志的重复记录
2026-04-27 21:43:37 +08:00
ygxbnet 987f0236a9 refactor(uploader): 优化上传功能的上下文管理和并发控制
构建上传工具 / build-tool (push) Successful in 1m16s
- 在 UploadDataToServer 函数中添加 context 参数支持
- 使用 http.NewRequest 替换 httpClient.Post 以更好地控制请求上下文
- 重构应用启动逻辑,在 StartUpload 中初始化上传器上下文
- 优化 StopUpload 方法中的上下文取消机制
- 移除上传过程中的 wg.Wait() 调用以改善并发性能
2026-04-27 13:38:14 +08:00
ygxbnet d44efeef8d fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Successful in 1m14s
2026-04-27 13:03:12 +08:00
ygxbnet 68564b7b80 fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Successful in 5m54s
2026-04-27 12:54:07 +08:00
ygxbnet f4c9228b2a fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Successful in 2m32s
2026-04-27 01:31:18 +08:00
ygxbnet e2dc7028df fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Failing after 3m23s
2026-04-27 01:24:29 +08:00
ygxbnet 9fb4817b6c fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Successful in 2m6s
2026-04-27 01:15:38 +08:00
ygxbnet 4233619fb3 fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Successful in 2m8s
2026-04-27 01:12:20 +08:00
ygxbnet 455eb7276d Merge remote-tracking branch 'origin/main'
构建上传工具 / build-tool (push) Successful in 3m30s
2026-04-27 01:06:27 +08:00
ygxbnet 1916b5cf54 fix(build): 修复构建工具配置问题 2026-04-27 01:06:07 +08:00
ygxbnet b544ba5a1b fix(build): 修复构建工具配置问题 2026-04-27 01:06:00 +08:00
ygxbnet 34834c478f fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Failing after 3m32s
2026-04-27 01:01:23 +08:00
ygxbnet 0007a80328 fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Failing after 3m34s
2026-04-27 00:54:11 +08:00
ygxbnet 8912ec7f9a fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Failing after 2m29s
2026-04-27 00:46:59 +08:00
ygxbnet 978f870bab fix(build): 修复构建工具配置问题
构建上传工具 / build-tool (push) Failing after 1m38s
2026-04-27 00:43:40 +08:00
13 changed files with 687 additions and 231 deletions
@@ -2,7 +2,7 @@ name: 构建上传工具
on: [ push ] on: [ push ]
jobs: jobs:
build-tool: build:
env: env:
RUNNER_TOOL_CACHE: /toolcache RUNNER_TOOL_CACHE: /toolcache
runs-on: ubuntu-latest runs-on: ubuntu-latest
@@ -10,8 +10,18 @@ jobs:
- name: 检出代码 - name: 检出代码
uses: actions/checkout@v4 uses: actions/checkout@v4
- name: 缓存依赖
uses: actions/cache@v4
with:
path: |
~/go/pkg/mod
~/.cache/go-build
/var/cache/apt
key: ${{ runner.os }}-go
- name: 安装Go(镜像) - name: 安装Go(镜像)
run: | run: |
set -x
echo "正在检查 Go 语言最新版本..." echo "正在检查 Go 语言最新版本..."
LATEST_GO_VERSION=$(curl -s https://go.dev/VERSION?m=text | head -n 1) LATEST_GO_VERSION=$(curl -s https://go.dev/VERSION?m=text | head -n 1)
if [ -z "$LATEST_GO_VERSION" ]; then if [ -z "$LATEST_GO_VERSION" ]; then
@@ -24,33 +34,33 @@ jobs:
sudo rm -rf /usr/local/go sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf go.tar.gz sudo tar -C /usr/local -xzf go.tar.gz
echo "/usr/local/go/bin" >> $GITHUB_PATH echo "/usr/local/go/bin" >> $GITHUB_PATH
echo "~/go/bin" >> $GITHUB_PATH
env: env:
GOROOT: /usr/local/go GOROOT: /usr/local/go
- name: 安装MinGW-w64 - name: 安装NodeJS
run: | uses: actions/setup-node@v6
apt update
apt install -y mingw-w64
x86_64-w64-mingw32-gcc --version
- name: 缓存依赖
uses: actions/cache@v4
with: with:
path: | node-version-file: 'frontend/package.json'
~/go/pkg/mod
~/.cache/go-build - name: 安装构建工具
key: ${{ runner.os }}-go run: |
set -x
npm install -g pnpm
go install github.com/wailsapp/wails/v2/cmd/wails@latest
- name: 构建上传工具 - name: 构建上传工具
run: | run: |
# go env -w CC=x86_64-w64-mingw32-gcc set -x
wails build git_hash=$(git rev-parse --short "$GITHUB_SHA")
-platform windows/amd64 build_date=$(TZ=Asia/Shanghai date +"%Y%m%d%H%M")
-ldflags "-X 'main.version=$(TZ=Asia/Shanghai date +"%m%d%H%M")'" wails build \
-platform windows/amd64 \
-ldflags "-X 'main.version=$build_date - $git_hash'" \
-o 上传工具.exe -o 上传工具.exe
- name: 上传构建文件 - name: 上传构建文件
uses: actions/upload-artifact@v3 uses: actions/upload-artifact@v3
with: with:
name: 上传工具 name: 上传工具
path: 上传工具.exe path: build/bin/上传工具.exe
+24 -13
View File
@@ -16,10 +16,9 @@ type App struct {
logChan chan string logChan chan string
uploaderCTX context.Context uploaderCTX context.Context
uploaderCancel context.CancelFunc uploaderCancel context.CancelFunc
isRun bool
} }
var isRun = false
func NewApp() *App { func NewApp() *App {
return &App{} return &App{}
} }
@@ -33,16 +32,25 @@ func (a *App) startup(ctx context.Context) {
go func() { go func() {
for log := range a.logChan { for log := range a.logChan {
runtime.EventsEmit(a.ctx, "log", log) runtime.EventsEmit(a.ctx, "log", log)
time.Sleep(time.Millisecond)
}
}()
// 后台 goroutine 持续推送运行状态
go func() {
for {
time.Sleep(250 * time.Millisecond)
runtime.EventsEmit(a.ctx, "is-run", a.isRun)
} }
}() }()
//在程序启动时运行上传程序 //在程序启动时运行上传程序
a.uploaderCTX, a.uploaderCancel = context.WithCancel(a.ctx) //if config.APPConfig.IsRunOnStart {
if config.APPConfig.IsRunOnStart { // time.Sleep(time.Second)
time.Sleep(time.Second) // a.uploaderCTX, a.uploaderCancel = context.WithCancel(a.ctx)
isRun = true // go uploader.StartUpload(a.uploaderCTX, &a.logChan)
go uploader.StartLooking(a.uploaderCTX, &a.logChan, config.APPConfig.CheckDir) // a.isRun = true
} //}
} }
// SelectPath 打开选择路径弹框 // SelectPath 打开选择路径弹框
@@ -57,20 +65,23 @@ func (a *App) GetConfig() config.Config {
} }
func (a *App) WriteConfig(key string, value any) { func (a *App) WriteConfig(key string, value any) {
fmt.Println(key, value) fmt.Println("写入配置:", key, value)
config.WriteConfig(key, value) config.WriteConfig(key, value)
} }
func (a *App) StartUpload() { func (a *App) StartUpload() {
if isRun { if a.isRun {
return return
} }
go uploader.StartLooking(a.uploaderCTX, &a.logChan, config.APPConfig.CheckDir) a.uploaderCTX, a.uploaderCancel = context.WithCancel(a.ctx)
go uploader.StartUpload(a.uploaderCTX, &a.logChan)
a.isRun = true
} }
func (a *App) StopUpload() { func (a *App) StopUpload() {
if isRun { if a.isRun {
a.uploaderCancel() a.uploaderCancel()
a.uploaderCTX, a.uploaderCancel = context.WithCancel(a.ctx)
} }
a.isRun = false
uploader.AddLog(&a.logChan, "上传程序已退出")
} }
+1
View File
@@ -11,6 +11,7 @@
"type-check": "vue-tsc --build" "type-check": "vue-tsc --build"
}, },
"dependencies": { "dependencies": {
"@element-plus/icons-vue": "^2.3.2",
"element-plus": "^2.13.7", "element-plus": "^2.13.7",
"vue": "^3.5.32" "vue": "^3.5.32"
}, },
+1 -1
View File
@@ -1 +1 @@
aa914e6b4676ee4621ced7ad6d81c58c 05225657934ff66d822c925754c951bf
+3
View File
@@ -8,6 +8,9 @@ importers:
.: .:
dependencies: dependencies:
'@element-plus/icons-vue':
specifier: ^2.3.2
version: 2.3.2(vue@3.5.33(typescript@6.0.3))
element-plus: element-plus:
specifier: ^2.13.7 specifier: ^2.13.7
version: 2.13.7(typescript@6.0.3)(vue@3.5.33(typescript@6.0.3)) version: 2.13.7(typescript@6.0.3)(vue@3.5.33(typescript@6.0.3))
+114 -70
View File
@@ -1,22 +1,28 @@
<script lang="ts" setup> <script lang="ts" setup>
import {ref, nextTick, watch} from 'vue' import {computed, nextTick, ref} from 'vue'
import {ElMessage} from 'element-plus' import {ElMessage, ElMessageBox} from 'element-plus'
import {ElMessageBox} from 'element-plus' import ConfirmClearDialog from './components/ConfirmClearDialog.vue'
import {SelectPath, GetConfig, WriteConfig, StartUpload, StopUpload} from '../wailsjs/go/main/App'; import {GetConfig, SelectPath, StartUpload, StopUpload, WriteConfig} from '../wailsjs/go/main/App';
import {config} from "../wailsjs/go/models.ts"; import {config} from "../wailsjs/go/models.ts";
import Config = config.Config;
import {EventsOn, LogPrint} from "../wailsjs/runtime"; import {EventsOn, LogPrint} from "../wailsjs/runtime";
import {configModel} from "@/model.ts";
import Config = config.Config;
const serverUrl = ref('') // const serverUrl = ref('')
const token = ref('') const token = ref('')
const checkDir = ref('') const checkDir = ref('')
const concurrentFiles = ref(1) const concurrentFiles = ref(1)
const uploadThreads = ref(1) const uploadThreads = ref(1)
// const autoStart = ref(false)
const progress = ref(0)
const isRunning = ref(false) const isRunning = ref(false)
const logOutput = ref<string[]>([]) const logOutput = ref<string[]>([])
const logContentRef = ref<HTMLElement>() const logContentRef = ref<HTMLElement>()
const logRoll = ref(true)
const clearDialogVisible = ref(false)
const filesToClear = ref<string[]>([])
const noPromptClear = ref(false)
interface FileProgress { interface FileProgress {
name: string name: string
@@ -25,14 +31,24 @@ interface FileProgress {
percentage: number percentage: number
} }
const progressList = ref<FileProgress[]>([ // {name: '测试文件1.txt', uploaded: 100, total: 500, percentage: 20},
// {name: '测试文件1.txt', uploaded: 100, total: 500, percentage: 20}, const progressList = ref<FileProgress[]>([])
])
const sortedProgressList = computed(() => {
return [...progressList.value].sort((a, b) => {
const getPriority = (item: FileProgress) => {
if (item.percentage === 100) return 2
if (item.percentage === 0) return 1
return 0
}
return getPriority(a) - getPriority(b)
})
})
const addLog = (msg: string) => { const addLog = (msg: string) => {
logOutput.value.push(`[${new Date().toLocaleString()}]` + msg) logOutput.value.push(`[${new Date().toLocaleString()}] ` + msg)
nextTick(() => { nextTick(() => {
if (logContentRef.value) { if (logContentRef.value && logRoll.value) {
logContentRef.value.scrollTop = logContentRef.value.scrollHeight logContentRef.value.scrollTop = logContentRef.value.scrollHeight
} }
}) })
@@ -41,43 +57,36 @@ const addLog = (msg: string) => {
const selectDirectory = () => { const selectDirectory = () => {
// ElMessage.info('请手动输入检测目录路径') // ElMessage.info('请手动输入检测目录路径')
SelectPath().then((path) => { SelectPath().then((path) => {
checkDir.value = path if (path) {
checkDir.value = path
} else {
ElMessage.warning('未选择目录,不更改配置')
}
}) })
} }
const startRun = () => { const startRun = () => {
if (!serverUrl.value) { // if (!serverUrl.value) {
ElMessage.warning('请输入服务器地址') // ElMessage.warning('请输入服务器地址')
return // return
} // }
if (!token.value) { if (!token.value) {
ElMessage.warning('请输入Token') ElMessage.error('请输入Token')
return return
} }
if (!checkDir.value) { if (!checkDir.value) {
ElMessage.warning('请选择检测目录') ElMessage.error('请选择检测目录')
return return
} }
isRunning.value = true
progress.value = 0
addLog("===============================================")
// addLog(`开始运行...`)
addLog(`服务器: ${serverUrl.value}`)
addLog(`检测目录: ${checkDir.value}`)
addLog(`同时处理文件数: ${concurrentFiles.value}`)
addLog(`单文件上传线程: ${uploadThreads.value}`)
addLog("===============================================")
StartUpload() StartUpload()
} }
const stopRun = () => { const stopRun = () => {
if (isRunning.value) { addLog(`正在停止运行`)
isRunning.value = false StopUpload().then(() => {
addLog(`正在停止运行`) ElMessage.success('已停止运行')
StopUpload().then(() => { })
ElMessage.info('已停止运行')
})
}
} }
const clearLog = () => { const clearLog = () => {
@@ -92,14 +101,35 @@ const clearLog = () => {
}) })
} }
// const writeServerUrl =() => {
// WriteConfig("url", serverUrl.value)
// }
const writeToken = () => {
WriteConfig(configModel.Token, token.value)
}
const writeCheckDir = () => {
WriteConfig(configModel.CheckDir, checkDir.value)
}
const writeConcurrentFiles = () => {
WriteConfig(configModel.HandleFileCount, concurrentFiles.value)
}
const writeUploadThreads = () => {
WriteConfig(configModel.ThreadCount, uploadThreads.value)
}
// const writeAutoStart = () => {
// WriteConfig("is-run-on-start", autoStart.value)
// }
// 加载配置 // 加载配置
try { try {
GetConfig().then((config: Config) => { GetConfig().then((config: Config) => {
serverUrl.value = config.url // serverUrl.value = config.url
token.value = config.token token.value = config.token
checkDir.value = config.check_dir checkDir.value = config.check_dir
concurrentFiles.value = config.handle_file_count concurrentFiles.value = config.handle_file_count
uploadThreads.value = config.thread_count uploadThreads.value = config.thread_count
// autoStart.value = config.is_run_on_start
noPromptClear.value = config.clear_files_no_prompt
LogPrint(`[${new Date().toLocaleString()}] 配置已加载`) LogPrint(`[${new Date().toLocaleString()}] 配置已加载`)
}) })
@@ -107,65 +137,68 @@ try {
console.log(e) console.log(e)
} }
watch(serverUrl, () => { try {
WriteConfig("url", serverUrl.value) EventsOn("is-run", (run) => {
}) isRunning.value = run
watch(token, () => { })
WriteConfig("token", token.value) EventsOn("progress", (progress) => {
}) progressList.value = progress
watch(checkDir, () => { })
WriteConfig("check-dir", checkDir.value) EventsOn("log", (msg) => {
}) addLog(msg)
watch(concurrentFiles, () => { })
WriteConfig("handle-file-count", concurrentFiles.value) EventsOn("clear-files", (files) => {
}) filesToClear.value = files
watch(uploadThreads, () => { if (!noPromptClear.value) {
WriteConfig("thread-count", uploadThreads.value) clearDialogVisible.value = true
}) }
})
EventsOn("log", (msg) => { } catch (e) {
addLog(msg) console.log(e)
}) }
EventsOn("progress", (progress) => {
progressList.value = progress
})
</script> </script>
<template> <template>
<div class="container"> <div class="container">
<div class="left-panel"> <div class="left-panel">
<div class="form-item"> <!-- <div class="form-item">-->
<label>服务器地址</label> <!-- <label>服务器地址</label>-->
<el-input v-model="serverUrl" placeholder="请输入服务器地址" :disabled="isRunning"/> <!-- <el-input v-model="serverUrl" placeholder="请输入服务器地址" :disabled="isRunning" @change="writeServerUrl()"/>-->
</div> <!-- </div>-->
<div class="form-item"> <div class="form-item">
<label>Token</label> <label>Token</label>
<el-input v-model="token" placeholder="请输入Token" :disabled="isRunning"/> <el-input v-model="token" placeholder="请输入Token" :disabled="isRunning" @change="writeToken()"/>
</div> </div>
<div class="form-item"> <div class="form-item">
<label>检测目录</label> <label>检测目录</label>
<div class="dir-input"> <div class="dir-input">
<el-input v-model="checkDir" placeholder="请选择检测目录" :disabled="isRunning"/> <el-input v-model="checkDir" placeholder="请选择检测目录" :disabled="isRunning" @change="writeCheckDir()"/>
<el-button @click="selectDirectory" :disabled="isRunning">选择目录</el-button> <el-button @click="selectDirectory" :disabled="isRunning">选择目录</el-button>
</div> </div>
</div> </div>
<div class="form-item"> <div class="form-item">
<label>同时处理文件数</label> <label>同时处理文件数</label>
<el-input-number v-model="concurrentFiles" :min="1" :max="100" :disabled="isRunning"/> <el-input-number v-model="concurrentFiles" :min="1" :max="100" :disabled="isRunning"
@change="writeConcurrentFiles()"/>
</div> </div>
<div class="form-item"> <div class="form-item">
<label>单文件上传线程</label> <label>单文件上传线程</label>
<el-input-number v-model="uploadThreads" :min="1" :max="100" :disabled="isRunning"/> <el-input-number v-model="uploadThreads" :min="1" :max="100" :disabled="isRunning"
@change="writeUploadThreads()"/>
</div> </div>
<!-- <div class="form-item">-->
<!-- <el-checkbox v-model="autoStart" label="运行时自动启动上传" size="large" :disabled="isRunning" @change="writeAutoStart()"/>-->
<!-- </div>-->
<div class="form-item"> <div class="form-item">
<label>上传进度</label> <label>上传进度</label>
<div class="progress-list"> <div class="progress-list">
<div v-for="(item, index) in progressList" :key="index" class="progress-item"> <div v-for="(item, index) in sortedProgressList" :key="index" class="progress-item">
<span class="file-name">{{ item.name }}</span> <span class="file-name">{{ item.name }}</span>
<el-progress :percentage="item.percentage" :status="item.percentage === 100 ? 'success' : undefined"/> <el-progress :percentage="item.percentage" :status="item.percentage === 100 ? 'success' : undefined"/>
<span class="progress-text">{{ item.uploaded }}/{{ item.total }}</span> <span class="progress-text">{{ item.uploaded }}/{{ item.total }}</span>
@@ -181,12 +214,20 @@ EventsOn("progress", (progress) => {
</div> </div>
<div class="right-panel"> <div class="right-panel">
<div class="log-header">日志输出</div> <div class="log-header">
日志输出
<el-checkbox v-model="logRoll" label="开启日志滚动"/>
</div>
<div class="log-content" ref="logContentRef"> <div class="log-content" ref="logContentRef">
<div v-for="(log, index) in logOutput" :key="index" class="log-line">{{ log }}</div> <div v-for="(log, index) in logOutput" :key="index" class="log-line">{{ log }}</div>
</div> </div>
</div> </div>
</div> </div>
<ConfirmClearDialog
v-model:visible="clearDialogVisible"
:file-list="filesToClear"
/>
</template> </template>
<style scoped> <style scoped>
@@ -212,7 +253,6 @@ EventsOn("progress", (progress) => {
.form-item { .form-item {
display: flex; display: flex;
flex-direction: column; flex-direction: column;
gap: 8px;
} }
.form-item label { .form-item label {
@@ -238,7 +278,8 @@ EventsOn("progress", (progress) => {
} }
.progress-list { .progress-list {
max-height: 160px; margin-top: 12px;
max-height: 250px;
overflow-y: auto; overflow-y: auto;
display: flex; display: flex;
flex-direction: column; flex-direction: column;
@@ -286,6 +327,9 @@ EventsOn("progress", (progress) => {
font-size: 14px; font-size: 14px;
font-weight: 500; font-weight: 500;
color: #303133; color: #303133;
display: flex;
justify-content: space-between;
align-items: center;
} }
.log-content { .log-content {
@@ -0,0 +1,124 @@
<script lang="ts" setup>
import {ref} from 'vue'
import {WriteConfig} from "../../wailsjs/go/main/App";
import {configModel} from "@/model.ts";
import {EventsEmit} from "../../wailsjs/runtime";
import {InfoFilled} from '@element-plus/icons-vue'
const props = defineProps<{
visible: boolean
fileList: string[]
}>()
const emit = defineEmits<{
(e: 'update:visible', value: boolean): void
// (e: 'confirm', dontShowAgain: boolean): void
}>()
const dontShowAgain = ref(false)
const handleClose = () => {
emit('update:visible', false)
EventsEmit('confirm-clear-files', false)
}
const handleConfirm = () => {
// emit('confirm', dontShowAgain.value)
emit('update:visible', false)
WriteConfig(configModel.ClearFilesNoPrompt, dontShowAgain.value)
EventsEmit('confirm-clear-files', true)
}
</script>
<template>
<el-dialog
:model-value="visible"
width="600px"
:close-on-click-modal="false"
@update:model-value="(val: boolean) => emit('update:visible', val)"
@close="handleClose"
align-center
>
<template #header>
是否确认清空需要上传文件
</template>
<div class="hint-text">以下文件将会被清空并移动到 tmp 文件夹进行上传,您是否确认</div>
<div class="dialog-content">
<div class="file-list">
<div v-for="(file, index) in fileList" :key="index" class="file-item">
{{ file }}
</div>
<div v-if="fileList.length === 0" class="empty-text">暂无文件</div>
</div>
</div>
<template #footer>
<div class="dialog-footer">
<div class="dialog-footer-checkbox-container">
<el-checkbox v-model="dontShowAgain" label="下次清空文件不再弹出此弹窗确认"/>
<el-tooltip content="此弹窗会在首次运行时弹出您可以选择下次不再弹出此弹窗">
<el-icon>
<InfoFilled/>
</el-icon>
</el-tooltip>
</div>
<div>
<el-button @click="handleClose">取消</el-button>
<el-button type="primary" @click="handleConfirm">确认</el-button>
</div>
</div>
</template>
</el-dialog>
</template>
<style scoped>
.dialog-footer-checkbox-container {
display: flex;
align-items: center;
gap: 5px;
}
.dialog-footer-checkbox-container .el-icon {
color: #909399;
}
.dialog-content {
padding: 10px 0;
}
.hint-text {
font-size: 12px;
color: #909399;
}
.file-list {
max-height: 350px;
overflow-y: auto;
border: 1px solid #e4e7ed;
border-radius: 4px;
padding: 8px;
}
.file-item {
padding: 6px 8px;
font-size: 14px;
color: #606266;
border-bottom: 1px solid #f0f0f0;
}
.file-item:last-child {
border-bottom: none;
}
.empty-text {
text-align: center;
color: #909399;
padding: 20px 0;
}
.dialog-footer {
display: flex;
justify-content: space-between;
align-items: center;
}
</style>
+9
View File
@@ -0,0 +1,9 @@
export enum configModel {
Url = "url",
Token = "token",
ThreadCount = "thread-count",
HandleFileCount = "handle-file-count",
IsRunOnStart = "is-run-on-start",
CheckDir = "check-dir",
ClearFilesNoPrompt = "clear-files-no-prompt",
}
+2
View File
@@ -7,6 +7,7 @@ export namespace config {
handle_file_count: number; handle_file_count: number;
is_run_on_start: boolean; is_run_on_start: boolean;
check_dir: string; check_dir: string;
clear_files_no_prompt: boolean;
static createFrom(source: any = {}) { static createFrom(source: any = {}) {
return new Config(source); return new Config(source);
@@ -20,6 +21,7 @@ export namespace config {
this.handle_file_count = source["handle_file_count"]; this.handle_file_count = source["handle_file_count"];
this.is_run_on_start = source["is_run_on_start"]; this.is_run_on_start = source["is_run_on_start"];
this.check_dir = source["check_dir"]; this.check_dir = source["check_dir"];
this.clear_files_no_prompt = source["clear_files_no_prompt"];
} }
} }
+56 -9
View File
@@ -1,37 +1,84 @@
package api package api
import ( import (
"context"
"dypid-client/internal/config" "dypid-client/internal/config"
"fmt"
"io" "io"
"net/http" "net/http"
"net/url" "net/url"
"sync"
"time" "time"
) )
var httpClient = &http.Client{ var httpClient = &http.Client{
Transport: &http.Transport{ Transport: &http.Transport{
MaxIdleConns: 200, MaxIdleConns: 500,
MaxIdleConnsPerHost: 200, MaxIdleConnsPerHost: 500,
IdleConnTimeout: 30 * time.Second, IdleConnTimeout: 30 * time.Minute,
}, },
Timeout: 30 * time.Second, Timeout: 30 * time.Second,
} }
func UploadDataToServer(data string) error { var limit chan struct{}
func init() {
//限制同时请求数为500
limit = make(chan struct{}, 500)
}
// InitConn 创建连接池
func InitConn() {
wg := &sync.WaitGroup{}
for i := 0; i < 10; i++ {
wg.Go(func() {
for i := 0; i < 50; i++ {
resp, err := httpClient.Get(config.APPConfig.Url + "/api/test")
if err != nil {
fmt.Println(err)
return
}
defer func() {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
}()
}
})
}
wg.Wait()
}
func UploadDataToServer(ctx context.Context, data string) error {
limit <- struct{}{}
defer func() {
<-limit
}()
params := url.Values{} params := url.Values{}
params.Set("token", config.APPConfig.Token) params.Set("token", config.APPConfig.Token)
params.Set("data", data) params.Set("data", data)
//http://127.0.0.1:8080/api/data?token=123456&data=123456 //http://127.0.0.1:8080/api/data?token=123456&data=123456
resp, err := httpClient.Post(config.APPConfig.Url+"/api/data?"+params.Encode(), request, err := http.NewRequest(
"", nil, "POST",
config.APPConfig.Url+"/api/data?"+params.Encode(),
nil,
) )
if err != nil { if err != nil {
return err return err
} }
if resp != nil { request.WithContext(ctx)
_, _ = io.Copy(io.Discard, resp.Body)
resp.Body.Close() resp, err := httpClient.Do(request)
if err != nil {
return err
} }
defer func() {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
}()
return err return err
} }
+35 -18
View File
@@ -2,38 +2,53 @@ package config
import ( import (
"fmt" "fmt"
"sync"
"github.com/fsnotify/fsnotify" "github.com/fsnotify/fsnotify"
"github.com/spf13/viper" "github.com/spf13/viper"
) )
type Config struct { type Config struct {
Url string `json:"url" mapstructure:"url"` Url string `json:"url" mapstructure:"url"`
Token string `json:"token" mapstructure:"token"` Token string `json:"token" mapstructure:"token"`
ThreadCount int `json:"thread_count" mapstructure:"thread-count"` ThreadCount int `json:"thread_count" mapstructure:"thread-count"`
HandleFileCount int `json:"handle_file_count" mapstructure:"handle-file-count"` HandleFileCount int `json:"handle_file_count" mapstructure:"handle-file-count"`
IsRunOnStart bool `json:"is_run_on_start" mapstructure:"is-run-on-start"` IsRunOnStart bool `json:"is_run_on_start" mapstructure:"is-run-on-start"`
CheckDir string `json:"check_dir" mapstructure:"check-dir"` CheckDir string `json:"check_dir" mapstructure:"check-dir"`
ClearFilesNoPrompt bool `json:"clear_files_no_prompt" mapstructure:"clear-files-no-prompt"`
} }
var APPConfig Config var APPConfig Config
var configMu sync.Mutex
const (
Url = "url"
Token = "token"
ThreadCount = "thread-count"
HandleFileCount = "handle-file-count"
IsRunOnStart = "is-run-on-start"
CheckDir = "check-dir"
ClearFilesNoPrompt = "clear-files-no-prompt"
)
func InitConfig() { func InitConfig() {
// 设置默认配置 // 设置默认配置
defaultConfig := Config{ defaultConfig := Config{
Url: "http://127.0.0.1:8080", Url: "http://127.0.0.1:8080",
Token: "", Token: "",
ThreadCount: 10, ThreadCount: 10,
HandleFileCount: 25, HandleFileCount: 25,
IsRunOnStart: false, IsRunOnStart: false,
CheckDir: "", CheckDir: "",
ClearFilesNoPrompt: false,
} }
viper.SetDefault("url", defaultConfig.Url) viper.SetDefault(Url, defaultConfig.Url)
viper.SetDefault("token", defaultConfig.Token) viper.SetDefault(Token, defaultConfig.Token)
viper.SetDefault("thread-count", defaultConfig.ThreadCount) viper.SetDefault(ThreadCount, defaultConfig.ThreadCount)
viper.SetDefault("handle-file-count", defaultConfig.HandleFileCount) viper.SetDefault(HandleFileCount, defaultConfig.HandleFileCount)
viper.SetDefault("is-run-on-start", defaultConfig.IsRunOnStart) viper.SetDefault(IsRunOnStart, defaultConfig.IsRunOnStart)
viper.SetDefault("looking-path", defaultConfig.CheckDir) viper.SetDefault(CheckDir, defaultConfig.CheckDir)
viper.SetDefault(ClearFilesNoPrompt, defaultConfig.ClearFilesNoPrompt)
//设置配置文件名和路径 ./config.toml //设置配置文件名和路径 ./config.toml
viper.AddConfigPath(".") viper.AddConfigPath(".")
@@ -61,6 +76,8 @@ func InitConfig() {
} }
func WriteConfig(key string, value any) { func WriteConfig(key string, value any) {
configMu.Lock()
viper.Set(key, value) viper.Set(key, value)
viper.WriteConfig() viper.WriteConfig()
configMu.Unlock()
} }
+289 -101
View File
@@ -7,20 +7,21 @@ import (
"dypid-client/internal/api" "dypid-client/internal/api"
"dypid-client/internal/config" "dypid-client/internal/config"
"fmt" "fmt"
"io"
"os" "os"
"path/filepath" "path/filepath"
"runtime"
"strconv"
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
"time" "time"
"github.com/wailsapp/wails/v2/pkg/runtime" wailsruntime "github.com/wailsapp/wails/v2/pkg/runtime"
"golang.org/x/sync/errgroup" "golang.org/x/sync/errgroup"
) )
var progress sync.Map type fileInfo struct {
type Task struct {
FilePath string FilePath string
FileLines int FileLines int
} }
@@ -32,140 +33,300 @@ type Progress struct {
Percentage int `json:"percentage"` Percentage int `json:"percentage"`
} }
func StartLooking(ctx context.Context, logChan *chan string, lookingPath string) { var progress sync.Map
func StartUpload(ctx context.Context, logChan *chan string) {
AddLog(logChan, "===============================================")
AddLog(logChan, `服务器: `+config.APPConfig.Url)
AddLog(logChan, `Token: `+config.APPConfig.Token)
AddLog(logChan, `检测目录: `+config.APPConfig.CheckDir)
AddLog(logChan, `同时处理文件数: `+strconv.Itoa(config.APPConfig.HandleFileCount))
AddLog(logChan, `单文件上传线程: `+strconv.Itoa(config.APPConfig.ThreadCount))
AddLog(logChan, "===============================================")
AddLog(logChan, "正在创建连接池(连接池可避免首次大量上传时出现网络错误)")
api.InitConn()
AddLog(logChan, "创建连接池完成,开始运行程序")
progress.Clear()
//推送上传进度
go func() { go func() {
for { for {
time.Sleep(500 * time.Millisecond) select {
case <-ctx.Done():
return
default:
}
var pg []Progress var pg []Progress
progress.Range(func(key, value any) bool { progress.Range(func(_, value any) bool {
p := value.(Progress) pg = append(pg, value.(Progress))
pg = append(pg, p)
return true return true
}) })
runtime.EventsEmit(ctx, "progress", pg) wailsruntime.EventsEmit(ctx, "progress", pg)
time.Sleep(250 * time.Millisecond)
} }
}() }()
//开启上传程序
for { for {
uploadData(ctx, logChan, lookingPath) uploadData(ctx, logChan)
select { select {
case <-time.After(time.Minute):
case <-ctx.Done(): case <-ctx.Done():
AddLog(logChan, "上传程序已退出")
return return
case <-time.After(time.Minute):
} }
} }
} }
func uploadData(ctx context.Context, logChan *chan string, lookingPath string) { func uploadData(ctx context.Context, logChan *chan string) {
var path = "./"
if lookingPath != "" {
path = lookingPath
}
//获取文件列表
files, err := getTxtFiles(path)
if err != nil {
AddLog(logChan, "获取文件列表失败:"+err.Error())
return
}
if files == nil {
return
}
start := time.Now() start := time.Now()
// 获取检测目录
var checkPath = "./"
if config.APPConfig.CheckDir != "" {
checkPath = config.APPConfig.CheckDir
}
//要上传的文件路径字符串数组
var files []string
//先检测tmp目录有没有残余文件
os.Mkdir("./tmp", os.ModePerm)
tmpFiles, err := getTxtFiles("./tmp")
if err != nil {
AddLog(logChan, "获取 tmp 文件列表失败:"+err.Error())
}
//tmp有文件,优先上传(else:tmp没文件扫描指定文件夹,并复制文件到tmp)
if tmpFiles != nil {
AddLog(logChan, "当前 tmp 目录下还有未上传完成文件,将优先上传 tmp 目录文件")
files = tmpFiles
} else {
//tmp没文件,扫描指定文件夹
f, err := getTxtFiles(checkPath)
if err != nil {
AddLog(logChan, "获取文件列表失败:"+err.Error())
return
}
//指定文件夹没文件,退出函数
if f == nil {
return
}
//是否向用户提示清空文件,并复制文件到tmp
if config.APPConfig.ClearFilesNoPrompt {
//不用提示直接复制文件到tmp
for _, p := range f {
select {
case <-ctx.Done():
return
default:
}
err := copyFile(p, "./tmp/"+filepath.Base(p))
if err != nil {
AddLog(logChan, "复制文件失败:"+err.Error())
} else {
files = append(files, "./tmp/"+filepath.Base(p))
err := os.Truncate(p, 0)
if err != nil {
AddLog(logChan, "清空文件失败:"+err.Error())
}
}
}
} else {
//提示用户
wailsruntime.EventsEmit(ctx, "clear-files", f)
confirm := make(chan bool)
wailsruntime.EventsOn(ctx, "confirm-clear-files", func(optionalData ...any) {
confirm <- optionalData[0].(bool)
})
if <-confirm {
for _, p := range f {
select {
case <-ctx.Done():
return
default:
}
err := copyFile(p, "./tmp/"+filepath.Base(p))
if err != nil {
AddLog(logChan, "复制文件失败:"+err.Error())
} else {
files = append(files, "./tmp/"+filepath.Base(p))
err := os.Truncate(p, 0)
if err != nil {
AddLog(logChan, "清空文件失败:"+err.Error())
}
}
}
} else {
AddLog(logChan, "已取消上传,1分钟后再运行")
return
}
}
}
//检测到文件 //检测到文件
//统计文件行数 //统计文件行数
fileLines := make(map[string]int) var filesInfo = make(map[string]fileInfo)
AddLog(logChan, fmt.Sprintf("正在统计 %v 个文件行数", len(files)))
isAllEmpty := true isAllEmpty := true
AddLog(logChan, fmt.Sprintf("正在统计 %v 个文件行数", len(files)))
for _, filePath := range files { for _, filePath := range files {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return return
default: default:
file, err := os.Open(filePath)
if err != nil {
AddLog(logChan, "打开文件失败:"+err.Error())
}
// 使用 bufio.Scanner 逐行读取
scanner := bufio.NewScanner(file)
lineCount := 0
for scanner.Scan() {
lineCount++
}
err = file.Close()
if lineCount == 0 {
continue
}
fileLines[filepath.Base(filePath)] = lineCount
isAllEmpty = false
AddLog(logChan, fmt.Sprintf("%s 文件行数:%v", filepath.Base(filePath), lineCount))
} }
file, err := os.Open(filePath)
if err != nil {
AddLog(logChan, "打开文件失败:"+err.Error())
}
// 使用 bufio.Scanner 逐行读取
scanner := bufio.NewScanner(file)
lineCount := 0
for scanner.Scan() {
lineCount++
}
err = file.Close()
if lineCount == 0 {
continue
}
filesInfo[filepath.Base(filePath)] = fileInfo{
FilePath: filePath,
FileLines: lineCount,
}
isAllEmpty = false
AddLog(logChan, fmt.Sprintf("%s 文件行数:%v", filepath.Base(filePath), lineCount))
} }
if isAllEmpty { if isAllEmpty {
AddLog(logChan, "所有文件都为空,不进行上传") AddLog(logChan, "所有文件都为空,不进行上传")
return return
} }
//添加文件上传任务参数(文件路径,文件行数) //刷新文件上传进度
var tasks []Task progress.Clear()
for k, v := range fileLines { for fileName, info := range filesInfo {
tasks = append(tasks, Task{FilePath: path + "/" + k, FileLines: v}) progress.Store(fileName,
Progress{
FileName: fileName,
Total: info.FileLines,
Uploaded: 0,
Percentage: 0,
},
)
} }
// 使用 errgroup 控制同时处理的文件数,并开始上传文件任务 // 使用 errgroup 控制同时处理的文件数,并开始上传文件任务
g, egctx := errgroup.WithContext(ctx) g, egctx := errgroup.WithContext(ctx)
g.SetLimit(config.APPConfig.HandleFileCount) // 设置同时处理文件数 // 设置同时处理文件数
// 执行所有任务 g.SetLimit(config.APPConfig.HandleFileCount)
for _, task := range tasks { // 执行文件上传任务参数(文件路径,文件行数)
for fileName, info := range filesInfo {
select {
case <-egctx.Done():
return
default:
}
g.Go(func() error { g.Go(func() error {
select { select {
case <-egctx.Done(): case <-egctx.Done():
return egctx.Err() return egctx.Err()
default: default:
AddLog(logChan, "正在上传文件:"+filepath.Base(task.FilePath))
processFile(egctx, logChan, task.FilePath, task.FileLines)
//上传完成,清空文件
err := os.Truncate(task.FilePath, 0)
if err != nil {
AddLog(logChan, "清空文件失败:"+err.Error())
}
return nil
} }
AddLog(logChan, "正在上传文件:"+fileName)
processFile(egctx, logChan, info.FilePath, info.FileLines)
select {
case <-egctx.Done():
return egctx.Err()
default:
}
//上传完成,删除缓存文件
err := os.Remove(info.FilePath)
if err != nil {
AddLog(logChan, "删除缓存文件失败:"+err.Error())
}
return nil
}) })
} }
// 等待所有任务完成 select {
if err := g.Wait(); err != nil { case <-ctx.Done():
AddLog(logChan, fmt.Sprintf("任务执行出错: %v", err)) return
} else { default:
AddLog(logChan, "所有任务执行完成!")
} }
// 等待所有任务完成
g.Wait()
AddLog(logChan, "所有任务执行完成!")
AddLog(logChan, fmt.Sprintf("上传完成,耗时:%s", time.Since(start).String())) AddLog(logChan, fmt.Sprintf("上传完成,耗时:%s", time.Since(start).String()))
progress.Clear()
} }
// 获取目录中的所有txt文件 // copyFile 快速拷贝文件 src -> dst
func copyFile(src, dst string) error {
// 打开源文件
sourceFile, err := os.Open(src)
if err != nil {
return err
}
defer sourceFile.Close()
// 创建目标文件
destFile, err := os.Create(dst)
if err != nil {
return err
}
defer destFile.Close()
// 核心:最快拷贝,底层使用操作系统零拷贝技术
_, err = io.Copy(destFile, sourceFile)
if err != nil {
return err
}
// 强制刷入磁盘,保证数据完整
return destFile.Sync()
}
// 获取目录中的所有txt文件(文件大小为0的不返回)
func getTxtFiles(dir string) (txtFiles []string, err error) { func getTxtFiles(dir string) (txtFiles []string, err error) {
err = filepath.Walk(dir, func(path string, info os.FileInfo, err error) error { err = filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil { if err != nil {
return err return err
} }
// 只处理普通文件,跳过目录 // 跳过目录,只处理普通文件
if !info.Mode().IsRegular() { if !info.Mode().IsRegular() {
return nil return nil
} }
// 检查文件扩展名是否为.txt // 检查文件扩展名是否为.txt
if strings.ToLower(filepath.Ext(path)) == ".txt" { if strings.ToLower(filepath.Ext(path)) == ".txt" {
txtFiles = append(txtFiles, path) if info.Size() != 0 {
txtFiles = append(txtFiles, path)
}
} }
return nil return nil
@@ -174,8 +335,8 @@ func getTxtFiles(dir string) (txtFiles []string, err error) {
return txtFiles, err return txtFiles, err
} }
// processFile 处理每个文件
func processFile(ctx context.Context, logChan *chan string, filePath string, fileLines int) { func processFile(ctx context.Context, logChan *chan string, filePath string, fileLines int) {
var wg sync.WaitGroup
// 打开文件 // 打开文件
file, err := os.Open(filePath) file, err := os.Open(filePath)
if err != nil { if err != nil {
@@ -185,74 +346,101 @@ func processFile(ctx context.Context, logChan *chan string, filePath string, fil
defer file.Close() defer file.Close()
// 创建行通道 // 创建行通道
lines := make(chan string, 100) lines := make(chan string, 200)
var countLine int32 = 0 var countLine int32 = 0
// 创建指定个worker同时处理文件上传 // 创建指定个worker同时处理文件上传
for i := 0; i < config.APPConfig.ThreadCount; i++ { for i := 0; i < config.APPConfig.ThreadCount; i++ {
wg.Go(func() { select {
case <-ctx.Done():
close(lines)
return
default:
}
go func() {
processLines(ctx, logChan, &lines, i, filePath, &countLine) processLines(ctx, logChan, &lines, i, filePath, &countLine)
}) }()
} }
// 读取文件并发送到通道 // 读取文件并发送到通道
scanner := bufio.NewScanner(file) scanner := bufio.NewScanner(file)
go func() { go func() {
defer func() {
if r := recover(); r != nil {
_, f, l, _ := runtime.Caller(0)
fmt.Println("panic:", f+":"+strconv.Itoa(l), r)
}
}()
for scanner.Scan() { for scanner.Scan() {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return return
default: default:
lines <- scanner.Text()
} }
lines <- scanner.Text()
} }
}() }()
// 等待所有行处理完成并推送进度
for int(countLine) != fileLines { for int(countLine) != fileLines {
select { select {
case <-ctx.Done(): case <-ctx.Done():
close(lines) close(lines) //关闭processLines中的上传线程
wg.Wait()
return return
default: default:
progress.Store(filepath.Base(filePath),
Progress{FileName: filepath.Base(filePath),
Total: fileLines, Uploaded: int(countLine),
Percentage: int(float64(countLine)/float64(fileLines)*100) + 1,
})
time.Sleep(500 * time.Millisecond)
} }
}
close(lines) progress.Store(filepath.Base(filePath),
wg.Wait() Progress{
FileName: filepath.Base(filePath),
Total: fileLines,
Uploaded: int(countLine),
Percentage: int(float64(countLine) / float64(fileLines) * 100),
},
)
}
//上传完成,进度设为100
progress.Store(filepath.Base(filePath),
Progress{
FileName: filepath.Base(filePath),
Total: fileLines,
Uploaded: int(countLine),
Percentage: 100,
},
)
close(lines) //关闭processLines中的上传线程
if err := scanner.Err(); err != nil { if err := scanner.Err(); err != nil {
AddLog(logChan, fmt.Sprintf("读取文件 %s 错误: %v", filePath, err)) AddLog(logChan, fmt.Sprintf("读取文件 %s 错误: %v", filePath, err))
return return
} }
AddLog(logChan, fmt.Sprintf("文件【%s】处理完成,共处理 %d 行数据", filePath, countLine)) AddLog(logChan, fmt.Sprintf("文件【%s】处理完成,共处理 %d 行数据", filepath.Base(filePath), countLine))
} }
// processLines 处理接受到的每一行数据并上传(chan 管道接受数据)
func processLines(ctx context.Context, logChan *chan string, lines *chan string, workerID int, filePath string, countLine *int32) { func processLines(ctx context.Context, logChan *chan string, lines *chan string, workerID int, filePath string, countLine *int32) {
for line := range *lines { for line := range *lines {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return return
default: default:
// 跳过空行
if strings.TrimSpace(line) == "" {
continue
}
// 上传数据
if err := api.UploadDataToServer(line); err != nil {
AddLog(logChan, fmt.Sprintf("Worker %d (文件 %s): 上传失败: %v", workerID, filePath, err))
}
atomic.AddInt32(countLine, 1)
} }
// 跳过空行
if strings.TrimSpace(line) == "" {
continue
}
// 上传数据
if err := api.UploadDataToServer(ctx, line); err != nil {
AddLog(logChan, fmt.Sprintf("Worker %d (文件 %s): 上传失败: %v", workerID, filepath.Base(filePath), err))
}
atomic.AddInt32(countLine, 1)
} }
} }
// AddLog 添加日志
func AddLog(logChan *chan string, message string) { func AddLog(logChan *chan string, message string) {
*logChan <- message *logChan <- message
} }
+1 -1
View File
@@ -22,7 +22,7 @@ func main() {
// Create application with options // Create application with options
err := wails.Run(&options.App{ err := wails.Run(&options.App{
Title: "dypid-client - 版本" + version, Title: "dypid-client - 版本" + version,
Width: 1024, Width: 1024,
Height: 768, Height: 768,
AssetServer: &assetserver.Options{ AssetServer: &assetserver.Options{